I guess you refer to this hadith which is quoted in Sunan ibn Majah, Sunan an-Nasa-i, Jami' at-Tirmidhi, Musnad Ahmad and Mustadrak al-Hakim:
Narrated Abdullah bin Buraidah narrated from his father:
that the Messenger of Allah (ﷺ) said: "The covenant between us and them is the Salat, so whoever abandons it he has committed disbelief." (Also narrted on the authority of ibn 'Abbas and Anas.
In his sharh (commentary) of Sunan an-Nasa-i Imam as-Suyuti said: this is a castigation for the one who doesn't pray and an alarming for him, which says you are nearer to become a kafir than being a Muslim or that you may end up as a non-Muslim if you don't repent and pray. He also added a quote of Al-Bayahqi saying something like this could also be taken literally as not praying could be considered as a disbelief which should be punished by taking his life as Imam Ahmad concluded from this hadith taking it literally.
I remember that there was a discussion between Imam a-Shafi'i and Imam Ahmad on the matter where Imam a-Shafi'i could clearly show a contradiction in the argumentation of Imam Ahmad.
There's also an other stronger hadith which you may find in almost all Sunan collections and in Sahih Muslim with slightly different wordings:
It is narrated on the authority of Abu Zubair that he heard Jabir b. 'Abdullah saying. I heard the Messenger of Allah (may peace and blessings be upon him) observing this:
Between man and polytheism and unbelief is the abandonment of salat.
So the prevalent view of the Hanbali school is that, somebody who doesn't pray is kafir. This is the opinion of Imam Ahamad, Imam ibn al-Mubarak, Ishaq and some of the students of a-Shafi'i.
While Imam Malik, a-Shafi'i and some of his students say such a person is not kafir, but fasiq and he should be asked to repent, so if he/she repents and prays the matter is closed and it's up to Allah to punish or forgive him in the hereafter. But if he/she didn't repent and insisted in not to pray he should be punished by taking his life.
Imam abu Hanifa and some other Kufi scholars and al-Muzni (student of a-Shafi'i) said he is neither considered as kafir nor should the authorities take his life, but he should be jailed until he prays.
And here's a fatwa defending the view of Imam Ahamad and the salafis saying it is the only correct view.
You could also take a look in this thread (in Arabic) who deals with the matter.
And Allah knows best!