The hadith is authentic, and it narrates the story of one of the Rabbis accepting Islam upon the arrival of the Prophet ﷺ to Medina. The full hadith text is:
حدثنا أبو المغيرة قال حدثنا صفوان قال حدثنا عبد الرحمن بن جبير بن نفير عن أبيه عن عوف بن مالك قال
— NOTE: My own translation, so treat with care:
Narrated by Abu Al-Mugheera through Safwan through Abdul-Rahman ibn Jubayr through his father through 'Awf ibn Malik, who said:
انطلق النبي صلى الله عليه وسلم يوما وأنا معه حتى دخلنا كنيسة اليهود بالمدينة يوم عيد لهم فكرهوا دخولنا عليهم فقال لهم رسول الله صلى الله عليه وسلم: يا معشر اليهود أروني اثني عشر رجلا يشهدون أنه لا إله إلا الله وأن محمدا رسول الله يحبط الله عن كل يهودي تحت أديم السماء الغضب الذي غضب عليه
One day, the Prophet ﷺ and I went to a Jewish synagogue in Medina on one of their festivals. They hated that we made an appearance. The Prophet ﷺ said: "O' Jewish tribe, show me twelve men from among you who testify that there is no deity but Allah,and that Muhammad is his messenger, and Allah will spare all the Jews under the visible heaven from the wrath He has upon them."
قال: فأسكتوا ما أجابه منهم أحد، ثم رد عليهم فلم يجبه أحد، ثم ثلث فلم يجبه أحد
They remained silent; no one answered him. So he repeated what he said, but and no one answered. So he repeated it the third time, but no one answered.
فقال: أبيتم فوالله إني لأنا الحاشر، وأنا العاقب، وأنا النبي المصطفى، آمنتم أو كذبتم
So he said: "By Allah, you deny. I am the last prophet, I am the final prophet, and I am the chosen prophet, whether you believe or deny.
ثم انصرف وأنا معه حتى إذا كدنا أن نخرج نادى رجل من خلفنا: كما أنت يا محمد. قال: فأقبل. فقال ذلك الرجل: أي رجل تعلموني فيكم يا معشر اليهود؟ قالوا: والله ما نعلم أنه كان فينا رجل أعلم بكتاب الله منك، ولا أفقه منك، ولا من أبيك قبلك، ولا من جدك قبل أبيك
The Prophet ﷺ and I then turned to leave. As we were about to exit, a man from behind us called: "As you are, O' Muhammad." The man walked towards us, then said: "O' Jewish tribe, what sort of man am I?" They replied: "By Allah, we do not know of any other man among us who knows the Book of Allah [Torah] more than you, nor anyone more knowledgeable about jurisprudence than you, and your father before you, and your grandfather before your father."
قال: فإني أشهد له بالله أنه نبي الله الذي تجدونه في التوراة، قالوا: كذبت ثم ردوا عليه قوله وقالوا فيه شرا
The man said: "Then, by Allah, I testify for him [the Prophet ﷺ] that he is Allah's messenger that you find in the Torah." They said: "You lied." Then they retracted what they said about him earlier, and said bad things about him.
قال رسول الله صلى الله عليه وسلم: كذبتم لن يقبل قولكم، أما آنفا فتثنون عليه من الخير ما أثنيتم ولما آمن أكذبتموه، وقلتم فيه ما قلتم، فلن يقبل قولكم
The Prophet ﷺ said: "You lied; what you said is not acceptable. First, you praise him with many virtues, then when he believes [in me], you say what you say now? What you say is not accepted."
قال: فخرجنا ونحن ثلاثة رسول الله صلى الله عليه وسلم، وأنا وعبد الله بن سلام، وأنزل الله عز وجل فيه
قل أرأيتم إن كان من عند الله وكفرتم به وشهد شاهد من بني إسرائيل على مثله فآمن واستكبرتم إن الله لا يهدي القوم الظالمين
Then the three of us, the Prophet ﷺ and 'Abdullah ibn Salam and I, exited. Allah then revealed:
Say, "Have you considered: if the Qur'an was from Allah, and you disbelieved in it while a witness from the Children of Israel has testified to something similar and believed while you were arrogant ... ?" Indeed, Allah does not guide the wrongdoing people. [Qur'an 46:10]
— Musnad Ahmad » Musnad Al-Ansar » Hadith 23464
This hadith was documented in Musnad Ahmad, and with similar chain of narration in Sahih Ibn Hibban, Musnad Al-Shamiyyeen, Al-Mustadrak, among other books of hadith.
The chain of narration is 'Awf ibn Malik Al-Ashja'i Al-Ghatafani (Arabic: عوف بن مالك الأشجعي الغطفاني) » Jubayr ibn Nufayr (Arabic: جبير بن نفير) » Abdul-Rahman ibn Jubayr (Arabic: عبد الرحمن بن جبير) » Safwan ibn 'Amr (Arabic: صفوان بن عمرو) » Abu Al-Mugheera (Arabic: بو المغيرة). This chain of narration is continuous and entirely comprised of trusted people that were documented by Muslim (hence, the inclusion of the hadith in Al-Mustadrak). The hadith is considered authentic.