Here is an answer that can explain both what your friend said (Prophet (sas) brought only these 4 persons under his cloak and called them his Ahle Bayt) and what the verse mentioned by @Ali Rabah Abdessamad in his answer (the verse define as "ahl lbayt" the wifes of the prophet):

First, the verse is revealed to the prophet, characterizing the prophet wife's as "Ahl albayt".
Second, and because both the Arabic meaning of the term "Ahl albayt" and the verse don't integrate the 4 persons mentioned by your friend, then for this cause, the prophet wanted to "add" the 4 persons, in order to show us how important they are to him, and therefore how important they should be to us.

Now this misconception your friend had, resulted directly from the fact that he privileges the hadith text, that fits his objective, upon the quran text, which questions his opinion about the Prohet's wifes.