Interesting Question.
Where do these people get their fatwas from?
First thing to analyze is that the pagan Arabs didn't have any festival that resembled Holi. Nor did the Romans, Jews, Christians or Persians who travelled or lived in Makkah or Madina.
Hence, it's not mentioned any where in Quran or in any Sahih Hadith .
But I have heard why it is 'not recommended' to play Holi by some present day 'Indian' scholars:
1) Muslims should not imitate others:
It was narrated that ‘Abd-Allaah ibn ‘Umar said: The Prophet (peace
and blessings of Allaah be upon him) said: “Whoever imitates a people
is one of them.” (Abu Dawood)
2) To avoid inter mingling and touching the other gender:
In present day Holi gathering, there is a lot of interaction between both genders such as touching and applying colors to their friends, families and sometimes strangers. Islam discourages touching a non-mahram.
3) To avoid consumption of alcohol:
A majority of the Holi gatherings provide an alcoholic drink called Bhang. In earlier times it was restricted to only Hindu men but now is offered to all guest irrespective of gender or religion (age being the only barrier). Needless to say, bhang is prohibited in Islam.
As you see, there is a great chance of indulging oneself in sins (perhaps even unintentionally). Hence, it is advised against attending such gatherings.
As far as your office mates forcibly putting color on you, it is done without your will. And Allah is Just. Each action will be judged only as per it's intention.
I'd also suggest to explain to your office friends your reservations similar to how many of your Hindu friends might tell you why they don't eat meat due to religious reasons. There is no need to be shy in religion. And most non-muslims are rational human beings who understand and have empathy for others.