That person was wrong.
When you give money for Zakat and you have larger bill and you want the other person to give you some change (out of Zakat money), he is obliged to give you that money out of Zakat fund or otherwise. You are not getting any money from that Zakat, you are in fact replacing smaller bills with larger bills, ending up giving more money.
One should not make life of a Muslim difficult. It is your right that you ask for exact change. It is his duty to provide you the exact change, no matter how much pain he has to go through for this. At best he may refuse to take your larger bill but he cannot deny you money back.
Collecting Mosque Funds in mosque does allow taking money back
It is common that in mosques when the collect fund, people drop larger bills and in return pick smaller bills to balance what they want to give. I personally do not approve this. If I have a larger bill, I will drop the whole bill because I do not want anything back from Allah house. So basically you drop money and you money money yourself from the fund, they do not object on it and it is generally ok.
If you are doing that outside mosque, I think become even more important that you get your money back out of zakat money. While it is true that note of Zakat should not be misused in any way, in this case, it is not misused but it is put to a good use because otherwise it will burden a Muslim beyond his means.
Optional work around: Change Money via third person
Another options that he can do is take your money, give it to third person and ask for change. He gets what you want to pay and puts the rest int he fund but again he cannot make it an excuse that he has zakat money and he cannot give it to other person. There are work arounds even he is worried about this. But again I don't think it is an issue or should be an issue. If the money is extremely large that I suggest for him to seek his master approval for such exchange but for ordinary currency, it is just how business work.