It has been said about this topic that if you can't find any other work to live on except this one (which I assume is not true in your case as it is known to exist in poor countries and those with great political conflicts), then you can stay in your work with avoiding the paths that lead to committing sins.
You mentioned that you exchange handshakes which is totally unpermitted in Islam:
It was narrated that Ma’qil ibn Yassaar said: the Messenger of Allaah (PBUH) said:
For one of you to be stabbed in the head with an iron needle is better for him than that he should touch a woman who is not permissible for him.
Narrated by al-Tabaraani in al-Kabeer, 486. Shaykh al-Albaani said in Saheeh al-Jaami’, 5045, that this hadeeth is valid.
Moreover, I don't find the necessity of doing it. Why not skip that part by saying you are a Muslim? Most people know that Muslims don't shack hands with non-mahram women, so they will understand.
You also mentioned that you watch the match but not the girls who would be wearing revealing clothes. Even if you thought that you understand yourself better than anyone else by saying genuinely that you don't watch them with lust eyes, still, Allah knows you the best. Why wouldn't he when he is your creator, in the first place. Thus, if Allah ordered us to avoid the Shaitan footsteps as they eventually lead to disobedience, then who are we to say that that is not true?!
Surah Al-Baqarah (2) - Verse 168
In Arabic (the exact words):
وَّ لَا تَتَّبِعُوۡا خُطُوٰتِ الشَّیۡطٰنِ ؕ اِنَّہٗ لَکُمۡ عَدُوٌّ مُّبِیۡنٌ...
The translation in English:
…and do not follow the footsteps of the Shaitan; surely he is your open enemy.
I suggest you play with males-only teams and to not participate in competitions where would be girls teams participants.
For further reading about Shaytan paths: See here
For further reading about shaking hands with non-Mahrams: See here