Question: What does it mean for a prayer to be invalid or not accepted?
I think the question isn't focused enough. Therefore the answer would be general hoping you may specify what exactly troubles you!
Scholars have defined a few rules which they consider as necessary for a prayer (I mean salat not du'a=supplication, which can be a part of the prayer also) to be valid. I'd say that this qualification of validity doesn't say anything about whether a prayer might be accepted by Allah or not, it is only a statement about the "technical correctness" of the done act of worship. As we never know whether a worship will be accepted by Allah, but should always hope so (see also my answer here).
An overview on the rules of validity
Shortly those are the fard (obligatory) parts of ablution (wudu') and prayer and some other rulings. Most of these parts are directly deduced from the Quran, for example for ablution, but scholars may define others too for example the intention/niyyah is often if not always considered as a fard because of the hadith, some may add the order of the acts etc.. Beside this any position in the prayer like qiyam (Standing) qu'ud (sitting), ruku' (bowing) and sujud (prostration) can be found in the Quran, so they are fard!
Now there are other necessary rules on the location where one prays and the person who prays:
For example the direction of prayer one is asked to do his best to pray towards al-Kaaba (this is considered as fard) because of:
... So turn your face toward al-Masjid al-Haram. ... in 2:144, 2:149 and 2:150
Of course you are asked to be (ritually) clean (not only ablution but in case you visited a toilet before the wudu' your body parts being clean too, see for example and this case), be wearing (ritually) clean (tahir) clothes and the place you pray on should be ritually) clean. One could find evidences for that. What does ritually clean may mean: For example they shouldn't have impurities on them like alcohol, urin, excrement ... and some may add if some of it has been licked by a dog (some would add a pig also) or came in contact with his muzzle.
To conclude if you performed the obligatory (fard) acts according any madhab you should be on the safe side. But for example let's say you didn't wash any part of your body three times or didn't wash your hands (only) at the beginning of the wudu' or didn't know the correct du'a for ruku' and said something different your prayer still would be accepted, as you didn't miss an essential part of it (see also this Post on missing an "essential" part of a prayer and this one on mispronunciation). The only problem is that you may missed a reward. But there are cases -and those are primarily related to the rules of prayer itself- where one can "repair" his/her prayer with sujud a-sahw (prostration of forgetfulness).
Here's a fatwa explaining how to perform ablution.
Here a fatwa about the rules of prayer (Please be aware that this is a salafi view).