What you are referring to is what is called tankees تَنْكِيس. The majority of scholars define it by reading the verses of quran in the reverse order for example reciting a surah beginning with the last verse and ending with the first verse. And this is forbidden.
Some scholars even defined tankees as reading for example surat al-Kahf (18) before surat al-Baqaqara (2), but this view is rather weak as you may conclude when reading this statement of ibn Battaal in his comment of sahih al-Bukahri quoted by ibn Hajar al-'Asqalani in fath-al-Bari:
Ibn Battaal said: We do not know of anyone who regarded it as obligatory to recite the soorahs in order, whether during prayer or otherwise, rather it is permissible to recite al-Kahf before al-Baqarah, or al-Hajj before al-Kahf, for example. As for what is narrated from the salaf about it being forbidden to recite the Qur’aan out of order, what is meant by that is reciting a soorah from its end to its beginning. There was a group which did that in poetry, to demonstrate their precise memorization thereof. So the salaf ruled that this is not allowed with regard to the Qur’aan and it is haraam. Al-Qaadi ‘Iyaad said: The order of soorahs is not obligatory when reading or praying, or studying or teaching, hence Mus-hafs varied, but when the Mus-haf of ‘Uthmaan was written they wrote it in the order in which it appears now. Hence the order of the Mus-hafs of the Sahaabah varied… then he mentioned something similar to Ibn Battaal. End quote from Fath al-Baari. (taken from this fatwa)
Details regarding tankees and the validity of prayers
One could also go in details, especially when it comes to applying this in prayers:
- tankees of surats تنكيس السور: Is as described in my example above above reading a surah before an other which in the order of the mushaf is placed before it. In the prayer one could for example recite after al-Fatiha surat al-Ikhlass (112) in the first rak'a and surat al-Masad (111) in the second rak'a.
Scholars have different opinions and ruling about it in prayer while they are in consensus that the prayer is valid! So some say it is allowed (Imam a-Shafi'i and Ahmad in one of his statements) while other consider it as frowned upon (hanbalis, malikis).
- tankees of verses تنكيس الآيات: Is as described reciting for example Verse 11 of a surah before Verse 10 or 9 and reciting 9 or 10 after reciting Verse 11. In this the majority of scholars are in consensus that it is haram and say if somebody does it by intention his prayer would be considered as invalid. A few scholars say again it is frowned upon. But if this tankees happend unintentionally the prayer is considered as valid. As a couple of scholars consider it as makrooh (frowned upon) and not forbidden (haram).
- tankees of words تنكيس الكلمات: Is when we recite words of a verse in an inappropriate order so that the verse would change or would be meaningless. In this there's consensus of scholars that it is haram and a prayer peformed with such a recitation is invalid no matter if it was by intenion or unintentionally.
- tankees of letters تنكيس الحروف: Has the same ruling as tankees of words, here we would even change letters in one word so the words of a verse would become meaningless!
See also these fatawa:
Ruling on tankees when memorising Quran (EN)
Reciting Surahs and verses in a different order (EN)
Ruling on tankeesRuling on tankees in the prayer (AR)