In Arabic language,
- "Ahlu Al bayt" are the ones living in the bayt(house). Ahlu(people of)
- "Ahlu al-Rajul" are the family of Rajul(man)
- "Ahlu al Bayt" refers to the prophet Muhammed (PBUH)
scholars have differed on who are Ahlul-Bayt:
1- His wives and children ONLY, they used this aya from the quran as a proof
O wives of the Prophet, you are not like anyone among women. If you fear Allah , then do not be soft in speech [to men], lest he in whose heart is disease should covet, but speak with appropriate speech.
And abide in your houses and do not display yourselves as [was] the display of the former times of ignorance. And establish prayer and give zakah and obey Allah and His Messenger. Allah intends only to remove from you the impurity [of sin], O people of the [Prophet's] household, and to purify you with [extensive] purification. http://quran.com/33/32-33
2- Those who are not allowed to take Zakkat, whole Banu Hisham family and Banu Mutalib.
The proof has been taken from the hadith in sahih muslim, Here is an excerpt from it:
He exhorted (us) (to hold fast) to the Book of Allah and then said: The second are the members of my household I remind you (of your duties) to the members of my family. He (Husain) said to Zaid: Who are the members of his household? Aren't his wives the members of his family? Thereupon he said: His wives are the members of his family (but here) the members of his family are those for whom acceptance of Zakat is forbidden. And he said: Who are they? Thereupon he said: 'Ali and the offspring of 'Ali, 'Aqil and the offspring of 'Aqil and the offspring of Ja'far and the offspring of 'Abbas. Husain said: These are those for whom the acceptance of Zakat is forbidden. Zaid said: Yes.
3- All the followers of the prophet Muhammed (PBUH) are considered muslims. They referred to this verse as a proof:
And the Day the Hour appears [it will be said], "Make the people of Pharaoh enter the severest punishment." http://quran.com/40/46
The verse says "'Alu pharoh"(people of Pharaoh). and since we say "'Alu Muhammed" in saying Salwat (Allahuma Sali ala Muhammed wa Ali Muhamed). This means the followers o are also considered Muslims.
4- Only Ali, Hasan, Hussien and Fatimah (RA on all of them). They took the hadith of Aisha (RA) in shaih Muslim:
that God's Apostle (may peace be upon him) went out one morning wearing a striped cloak of the black camel's hair that there came Hasan b. 'Ali. He wrapped him under it, then came Husain and he wrapped him under it along with the other one (Hasan). Then came Fatima and he took her under it, then came 'Ali and he also took him under it and then said: God only desires to take away any uncleanliness from you, O people of the household, and purify you (thorough purifying)"
For more information and details: [ARABIC] Alu al Bayt - Othman al-Khamys
At the end, This difference of opinion doesn't change anything the love and respect muslims have toward Ahlulbayt. Actually the difference of opinion made me quite happy as the 3rd opinion says that all Muslims are Ahlu-Al Bayt. I know that allah is the most merciful. Hence, let's say I won't be surprised if on judgment day, we find out that all muslims are considered Alhu al Bayt.