We know that Ibrahim rebuilt/built the Ka'ba. And it was a place of worship for the prophets after him. Doesn't this mean Ka'ba was a place of worship for the Christians and Jews? That is should be revered among them as well?
Whether there are references to Mecca in previous scripture is a matter of debate, for which Islam Stack Exchange is not the right forum. Furthermore, any such references will not carry authoritative weight among the people of the scripture since we are not experts on their text (e.g., is the Valley of Bakkah mentioned in Qur'an 3:96 the same Valley of Baka mentioned in Psalm 84:6?)
Therefore, I will focus the answer on the Islamic narrations of other prophets and messengers visiting Mecca.
Imam Malik ibn Anas mentioned in his Muatta' a hadith by the Prophet ﷺ talking about 70 prophets being born in Mina (about 10 km east of the Kaaba):
وحدثني عن مالك، عن محمد بن عمرو بن حلحلة الديلي، عن محمد بن عمران الأنصاري، عن أبيه، أنه قال عدل إلى عبد الله بن عمر وأنا نازل، تحت سرحة بطريق مكة فقال ما أنزلك تحت هذه السرحة فقلت أردت ظلها. فقال هل غير ذلك فقلت لا ما أنزلني إلا ذلك. فقال عبد الله بن عمر قال رسول الله صلى الله عليه وسلم: إِذَا كُنْتَ بَيْنَ الأَخْشَبَيْنِ مِنْ مِنًى - وَنَفَخَ بِيَدِهِ نَحْوَ الْمَشْرِقِ - فَإِنَّ هُنَاكَ وَادِيًا يُقَالُ لَهُ السُّرَرُ بِهِ شَجَرَةٌ سُرَّ تَحْتَهَا سَبْعُونَ نَبِيًّا
Yahya related to me from Malik from Muhammad ibn Amr ibn Halhala ad-Dili from Muhammad ibn Imran al-Ansari that his father said that Abdullah ibn Umar came upon him while he stopped for a rest under a tall tree on the road to Makka, and he said, "What has made you stop under this tall tree?" He replied that he sought its shade. Abdullah ibn Umar said, "Anything besides that?" and he said, "No, that was the only. reason he stopped for a rest," and Abdullah ibn Umar said, "The Messenger of Allah, may Allah bless him and grant him peace, said, 'If you are between al-Akhshabayn (which are two mountains) near Mina,' indicating the east with his outspread hand, 'you will find a valley called as-Surar with a tree in it beneath which the umbilical cords of seventy prophets have been cut.'"
In another narration, Imam Muslim ibn al-Hajjāj documented a hadith in which the Prophet ﷺ described Mussa ﷺ and Yūnus ﷺ descending upon the valley of Mecca while doing talbiyah (calling upon Allah):
عن ابن عباس أن رسول الله صلى الله عليه وسلم مر بوأدي الأزرق فقال أَىُّ وَادٍ هَذَا؟ فقالوا هذا وأدي الأزرق. قال: أَنِّي أَنْظُرُ إِلَى مُوسَى - عَلَيْهِ السَّلاَمُ - هَابِطًا مِنَ الثَّنِيَّةِ وَلَهُ جُؤَارٌ إِلَى اللَّهِ بِالتَّلْبِيَةِ. ثم أتى على ثنية هرشى. فقال: أَىُّ ثَنِيَّةٍ هَذِهِ؟ قالوا ثنية هرشى. قال: كَأَنِّي أَنْظُرُ إِلَى يُونُسَ بْنِ مَتَّى - عَلَيْهِ السَّلاَمُ - عَلَى نَاقَةٍ حَمْرَاءَ جَعْدَةٍ عَلَيْهِ جُبَّةٌ مِنْ صُوفٍ خِطَامُ نَاقَتِهِ خُلْبَةٌ وَهُوَ يُلَبِّي
Abu al-'Aliya narrated it on the authority of Ibn 'Abbas that the Messenger of Allah (ﷺ) passed through the valley of Azraq, and he asked: Which valley is this? They said: This is the valley of Azraq, and he observed: (I perceive) as if I am seeing Moses (peace be upon him) coming down from the mountain track, and he is calling upon Allah loudly (saying: Here I am! at your service!). Then he came to the mountain track of Harsha. He (the Holy Prophet) said: Which is this mountain track? They said: It is the mountain track of Harsha. He observed (I feel) as If I am seeing Yunus (Jonah-peace be upon him) son of Matta on a well-built red dromedary, with a cloak of wool around him and the rein of his dromedary is made of the fibres of date-palm, and he is calling upon Allah (saying: Here I am! at your service, my Lord! ).
Another hadith in Sahih Muslim prophesied 'Issa ﷺ performing talbiyah (calling upon Allah) and hajj (pilgrimage) in the future (after his descent from above on his second coming to Earth):
عن حنظلة الأسلمي قال سمعت أبا هريرة - رضي الله عنه - يحدث عن النبي صلى الله عليه وسلم قال: وَالَّذِي نَفْسِي بِيَدِهِ لَيُهِلَّنَّ ابْنُ مَرْيَمَ بِفَجِّ الرَّوْحَاءِ حَاجًّا أَوْ مُعْتَمِرًا أَوْ لَيَثْنِيَنَّهُمَا
Hanzala al-Aslami reported: I heard Abu Huraira (Allah be pleased with him) as narrating from Allah's Apostle (ﷺ) who said: By Him in Whose Hand is my life. Ibn Maryam (Jesus Christ) would certainly pronounce Talbiya for Hajj or for Umra or for both (simultaneously as a Qiran) In the valley of Rauha.
There are several other athars by the companions (not attributed to the Prophet ﷺ) on the topic. As an example, in Ad-Durr al-Manthūr by Jalāl Ad-Dīn As-Suyuti:
عن عروة بن الزبير قال بلغني أن البيت وضع لآدم عليه السلام يطوف به ويعبد الله عنده وأن نوحا قد حجه وجاءه وعظمه قبل الغرق فلما أصاب الأرض من الغرق حين أهلك الله قوم نوح أصاب البيت ما أصاب الأرض فكان ربوة حمراء معروف مكانه فبعث الله هودا إلى عاد فتشاغل حتى هلك ولم يحجه ثم بوأه الله لإبراهيم عليه السلام فحجه وعلم مناسكه ودعا إلى زيارته ثم لم يبعث الله نبيا بعد إبراهيم إلا حجه
NOTE. My own translation, so treat with care.
Urwah ibn Zubayr narrated that he was informed that the House [Ka'bah] was put for Adam ﷺ so that he performs tawāf (walking around it) and worships Allah there, and that Nūh ﷺ performed hajj there, visited it, and glorified it prior to the flooding. When the Earth was hit with the flood, it got hit by what hit the Earth, and it became a known red elevated ground. Allah ﷻ sent Hūd to 'Aad, who kept him occupied until his death and he did not visit the place. Then Allah ﷻ designated for Ibrāhīm ﷺ its site. He was informed of its holy rites, and he supplicated and called for visiting it, and there is no prophet that Allah sent after Ibrāhīm ﷺ who has not performed hajj there.
— Ad-Durr al-Manthūr, Vol. 1, pp. 316 (Arabic only)
Yes. Believers from among the People of the Book respected it, protected it and did rites there.
For example, there is the story of Tubba (upon the religion of Moses) and how he was considering destroying the Kaaba [due to what he was seeing of Jahiliyyah there, not knowing its history or prophesies regarding it] and then two Jewish Rabbis stopped him from doing so and explained its significance. Ibn Kathir رحمه الله mentioned concerning Tubba:
...He was accompanied by two Jewish rabbis who advised him and told him that he would never prevail over this city (Madinah), for it would be the place to which a Prophet would migrate towards the end of time. So he retreated and took them (the two rabbis) with him to Yemen.
When he passed by Makkah, he wanted to destroy the Ka`bah, but they told him not to do that either. They told him about the significance of this House, that it had been built by Ibrahim Al-Khalil, peace be upon him, and that it would become of great importance through that Prophet who would be sent towards the end of time. So he respected it, performed Tawaf around it, and covered it with a fine cloth.. - Qur'an 44:37.
Notice, they associated a future Prophet with Madinah and Makkah (Kaaba). The People of the Book had been waiting for the promised Prophet in Arabia, hence an historical Jewish presence in Madinah. But when he finally came, many of them betrayed the truth.
Furthermore, IslamQA does a great job compiling verses from the Torah that demonstrate the connection of Abraham/his family to Makkah. Read here: https://islamqa.info/en/204341
