Your father must be asked first. If he refuses to arrange the marriage, then you might pass to the next wali in line.
If the wali repeatedly refuses the proposal of a compatible suitor, he
is to be regarded as preventing the marriage of the female relative
under his care, and his guardianship is thus rendered null and void,
and that right is transferred to the next closest relative on the
That being said, I don't know how Islam considers a wali who abandoned is ...wali-hood (?)... If your father never was a father to you, if he relinquished his right to be your father from your young age, I don't know if he can still be considered for the position of Wali. Maybe that's the question you should ask.