My dad has been physically and verbally abusive to my family my whole life, especially to my mom. He pretends to be nice at times but does it again and he always talks about how much he hates us. There’s no way to talk to him and make him a better person he’s evil. I hate him also and I plan to abandon him completely soon, and I try to convince my mom and siblings to come with me. Would it be a sin? Even if it is I still will but I just want to know.
In Islam, wife-beating, abusing, striking, and causing harm against others that are undeserving of it is absolutely haram.
'A'isha reported that Allah's Messenger (ﷺ) never beat anyone with his hand, neither a woman nor a servant, but only, in the case when he had been fighting in the cause of Allah and he never took revenge for anything unless the things made inviolable by Allah were made violable; he then took revenge for Allah, the Exalted and Glorious.
'Urwa b. Zubair reported that Hisham b. Hakim found a person (the ruler of Hims) who had been detaining some Nabateans in connection with the dues of Jizya. He said: What is this? I heard Allah's Messenger (ﷺ) as saying: Allah would torment those persons who torment people in the world.
You can read more here: https://www.abuaminaelias.com/prophet-never-beat-women-servants/
According to what you've said, you have tried to speak to him about his flaws and get him to try to change his ways but it was to no avail. It would be of no fault and actually praiseworthy for you to proceed with bringing your mother and siblings to live with you in order to protect them from harm.
I recommend you speak to a lawyer about this and bring this issue to court. Your mother's injuries should suffice as enough evidence for the courts to side with you. After that, using your best rationality, decide whether it would be ok to still be in contact with him.
What you are planning to do is not only a big sin but it will decrease your life span significantly. You are not Allah and today is not the judgment day. Ask the god for forgiveness (towbah nasooh). If you are going to do it anyway, make sure to check on him from time to time, at least without he knowing it. The god is watching you, so even if you hate him, don't really abandon him, this way you avoid direct communication and you still are taking care of him. He can say he hates you, but you should not. However no body other than God knows if he really hate you (not even himself). He might be mentally sick and abandoning him would be a big sin. Find somebody that he respect and ask him/her for helping you by convincing him to see a psychiatrist.