Some people tell me that I sometimes look away when I do salah, not move my head but like quickly look at my right side, etc. Will this invalidate my salah?
Looking around during salah without any genuine need is disliked (makrooh).
According to Ibn Qudaamah, it is mustahabb for the worshipper to fix his gaze on the place of prostration. Ahmad said – according to the report of Hanbal – : Proper focus and humility (khushoo’) in prayer means that (the worshipper) should fix his gaze on the place of prostration - [Al-Mughni, 1/370]
Narrated by Aisha (R.A):
I asked Allah's Apostle about looking hither and thither in prayer. He replied, "It is a way of stealing by which Satan takes away (a portion) from the prayer of a person." - [Sahih al-Bukhari Vol. 1, Book 12, Hadith 718]
Anas (r.a) relates that the Messenger of Allah (s.a.w.w) said to him:
“Be careful about turning during the salah as turning in the salah is disastrous. If you must do it, then do it in the voluntary prayers but not in the obligatory prayers.” [related by at-Tirmidhi who calls it sahih]
Hadrat Abu Dharr (r.a.) reported that the Prophet (saws) said:
"Allah faces the slave while he is in the salah and keeps facing him as long as he does not turn. If [the slave] turns, [Allah] turns away from him." -Related by Ahmad and by Abu Dawud.
So if you turned your gaze anywhere but prostration unknowingly or in case of emergency then I hope your salah will be validade but Allah know the best.