Obviously, it is not Haram if we were to go by the verses of the Quran and the following Ahadith.
It is narrated on the authority of Ibn Hibban, Al-Bazzar, Al-Tabari, and Ibn Mardawih that Umm `Atiyyah said in respect of the story of taking the oath of allegiance of women, “The Messenger of Allah (peace and blessings be upon him) held out his hand from outside the house and we (the immigrating women) held our hands from within the house, then he said, ‘O Allah, bear witness."
It is narrated on the authority of Anas ibn Malik (may Allah be pleased with him) that he said, “Any of the female slaves of Madinah could take hold of the hand of Allah's Messenger and take him wherever she wished.” (Reported by Al-Bukhari)
Furthermore, just because a woman happened to be a slave that doesn't mean she is any less of a woman than those who are not.
The context of the above reports become clear when we look at the following verses of the Quran and the report associated with Um Atiya
Indeed, those who pledge allegiance to you only they pledge allegiance (to) Allah. (The) Hand (of) Allah (is) over their hands. Then whoever breaks (his oath) then only he breaks against himself, and whoever fulfills what he has covenanted with Allah, soon He will give him a reward great Quran 48:10.
O Prophet! When come to you the believing women pledging to you [on] that not they will associate with Allah anything, and not they will steal, and not they will commit adultery, and not they will kill their children, and not they bring slander, they invent it between their hands and their feet, and not they will disobey you in (the) right, then accept their pledge and ask forgiveness for them (from) Allah. Indeed, Allah (is) Oft-Forgiving, Most Merciful. Quran 60:12
We took the oath of allegiance to Allah's Messenger (ﷺ) and he recited to us: 'They will not associate anything in worship with Allah,' and forbade us to bewail the dead. Thereupon a lady withdrew her hand, and said, "But such-and-such lady lamented over one of my relatives, so I must reward (do the same over the dead relatives of) hers." The Prophet (ﷺ) did not object to that, so she went (there) and returned to the Prophet (ﷺ) so he accepted her pledge of allegiance.
Bukkhari Book 65, Hadith 4892
Although the translation from the Arabic text is not 100% but the context is clear once again.
It is also clear from the verse Quran 48:10 above that no distinction is made between a man and a woman. And if we were to speculate that the mention of the hands is symbolic then it must be symbolic for both the genders. And Quran 60:12 makes it more than clear that the oath of allegiance was to be taken from the women folks as well and no mention of not shaking hand is added.
Those who adhere to the no shaking hand position mainly rely on two Ahadith, one single Hadith ascribed to Aisha and the other one is from Dawood. It is a weak position because as we all know the entire body of Ahadith claimed to have been narrated by Aisha is a time line quagmire. And here is why:
Firstly, it is a chronological impossibility to determine the age of Aisha from the Hadith literature without creating time-line contradictions. Although a separate discussion but a glimpse of the problem can be gauged from the fact that out of the total of 2210 reports associated with Aisha, Buhkari and Muslim could agree on a mere 174 in total. A shockingly low number.
Secondly, the time-line is a problem because it goes to show what Aisha could and could not have witnessed and when. These type of discrepancies could be glossed over in a bygone era but with the current tools at our disposal we have to be very careful.
The last thing we want to do is give ammunition to the enemies of Islam to discredit it.