We all know that Allah tests humans with various things. And my father yells at me for no reason. And he also yells at other members of my family. I yell at him back. But I saw some YouTube video sheikh saying that it is wrong and I have to be patient. After that I try to be patient as possible but some point I can't hold it in. I am still trying though but it's not good enough. So does that mean I have failed in the test of Allah? And it is a punishment for me? And I feel really bad now. And my faith is also slowing down.
Don't fret :) The test is ongoing for as long as you are alive, so you have many chances to repent and pass.
In the Qur'an, Allah (ﷻ) said not to say words of annoyance ("uff") against our parents. You have to be patient with them and try to speak to them in the best way, even when they make you annoyed or upset or angry. If you have spoken to them in anger in ways that Islam says in unsuitable, all you should do is repent.
Does that mean Islam wants us to keep being verbally abused and do nothing? No!
My father was similar to yours, may Allah (ﷻ) bless him. We as a family made a collaborated effort to get him to change. Any time he would start arguments for no reason, we'd all tag-team and teach him not to cause such angry outbursts. Having done this method for months into years, it started affecting his heart. He is a much better person now than he once was.
Maybe you can do a similar approach tailored for your father specifically? Or maybe you have some wise elders with whom he is close that can intervene and guide him, etc? My overall point is, take an action that is both Islamically acceptable and will change your situation; sitting quietly and letting him yell at you all will solve nothing and yelling back and forth with him also solves nothing.
Above all, don't let this affect your heart/faith. Be strong.
Patience is understood in the wrong way by most people, in a backward way. That is the "patience" of the weak. They think patience means to bear foolishness and oppression whereas the true patience of the righteous has always been to be patient in removing foolishness and oppression from the Earth (beginning in ever-widening concentric circles). This is the patience of strength.
You have the option, actually the obligation, of not remaining in an oppressive situation, so you can move out. Do not tolerate foolishness, as silence is the same as being complicit (well, in most cases anyway). There are extremely serious matters ahead of us, so wasting time and energy bickering with each other is not becoming of any human let alone a believer. If you cannot move out, then I suggest you address that instead of arguing. What will shouting back at him achieve ?