Consider the scenario: I am sitting on my bed in a room. Now some person enters the room and starts praying namaz (salah) on the floor. His position while praying is such that he is facing me. Should I get aside because I am like in front of a person praying namaz?
Well done, you inquired about a question (issue) which could happen plenty of times. So it doesn't look to be counted as the items which invalidate the saying prayer (salah), and actually it could be the duty of the person who intends to say prayers to observe the issue and attempt not to stand in front of you who you were there since before. But on the other hand, we can say that perhaps it could be considered as a positive practice for you to move a little in order to changing your situation in front of him as holding more respect to Allah.