The most basic and simple answer they used to conclude this rule is the hadith in Sahih Muslim :
Jabir (May Allah be pleased with him) reported:
The Messenger of Allah (PBUH) said, "Between a man and disbelief and paganism is the abandonment of Salat (prayer)."
Now, Here are more Hadiths that Imam Ahmed and others used to support their opinion :
Buraidah (May Allah be pleased with him) reported: Messenger of Allah (PBUH) said, "That which differentiates us from the disbelievers and hypocrites is our performance of Salat. He who abandons it, becomes a disbeliever."
It has been narrated on the authority of 'Auf b. Malik that the Messenger of Allah (may peace be upon him) said: The best of your rulers are those whom you love and who love you, who invoke God's blessings upon you and you invoke His blessings upon them. And the worst of your rulers are those whom you hate and who hate you and whom you curse and who curse you. It was asked (by those present): Shouldn't we overthrow them with the help of the sword? He said: No, as long as they establish prayer among you. If you then find anything detestable in them. You should hate their administration, but do not withdraw yourselves from their obedience.
There are verses in the Quran too. But since you are looking for Hadiths only then I will not post them in my answer.