Would Allah allow you to feed your baby something impure?
Milk is not among the things that break wudu': Which things break the wudu?
In verse (16:66) milk was declared as pure:
... We give you drink from what is in their bellies - between excretion and blood - pure milk, palatable to drinkers.
The scholars are in full consent that it is pure, here a statement of the hanbali scholar 'ala' ad-Dyn al-Mardaawi علاء الدين المرداوي:
Al-Mardaawi (may Allah have mercy on him):
Milk of humans and animals that may be eaten is taahir, and there is no scholarly conflict on this point.
End quote from al-Insaaf, 1/343 (source islamqa #167149)
You may find similar statements from scholars of other madhabs in this fatwa on islamweb (in Arabic)
Imam al-Qurtobi his tafsir even quotes a dispute about the purness of breast milk of a dead woman. While al-Jassass in his tafssir even discusses the purness of the breast milk of dead cattle saying that the fact that the verse above is a general declaration of pureness.