It is OK. It is a mis-conception that non Muslims aren't allowed to enter the Haram (nothing about Mecca). The quran says:
O you who have believed, indeed the polytheists are unclean, so let them not approach al-Masjid al-Haram after this, their [final] year. And if you fear privation, Allah will enrich you from His bounty if He wills. Indeed, Allah is Knowing and Wise.
As you see this clearly states the Masjid Alharam and not Mecca. The above Ayah is the only proof some scholars mention that you are not allowed to visit Mecca if you are not a person of the book (Muslim, Christian, Jew).
Additionally, the most important point people miss in regarding to this Ayah is the reason why it was sent. During the time of the prophet the disbelievers would visit the holy Masjid and even pray in it. This is why the Ayah was sent down to stop them from doing so.
It was reported in the Sunan Alnisay, Mosnad Imam Ahmed, and many others including Sahih Muslim and Bukhari that the prophet said when this Ayah was brought down:
لما نزلت الآية قال النبي صلى الله عليه وسلم لعلي : ناد في أذانك ألا يحج بعد العام مشرك
When this Ayah came down the Prophet PBUH said to Ali: Call upon the people that no disbeliever is allowed to make Hajj after this year.
As you can clearly see the prophet PBUH interpreted this Ayah as going disbelievers are not allowed to go perform Hajj to the Kabbah and not what is generally believed.
As a matter of fact, many disbelievers used to come to Mecca after this Ayah came down (trade wise), and the prophet never forbade them to not come in.
The only other Hadith is the one that the prophet PBUH advised (not ordered) his people on his deathbed. The complete list is here, but I'll post one of them (they all are the same pretty much):
حَدَّثَنَا سَعِيدُ بْنُ مَنْصُورٍ، حَدَّثَنَا سُفْيَانُ بْنُ عُيَيْنَةَ، عَنْ سُلَيْمَانَ الأَحْوَلِ، عَنْ سَعِيدِ بْنِ جُبَيْرٍ، عَنِ ابْنِ عَبَّاسٍ، أَنَّ رَسُولَ اللَّهِ صلى الله عليه وسلم أَوْصَى بِثَلاَثَةٍ فَقَالَ " أَخْرِجُوا الْمُشْرِكِينَ مِنْ جَزِيرَةِ الْعَرَبِ وَأَجِيزُوا الْوَفْدَ بِنَحْوٍ مِمَّا كُنْتُ أُجِيزُهُمْ " . قَالَ ابْنُ عَبَّاسٍ وَسَكَتَ عَنِ الثَّالِثَةِ أَوْ قَالَ فَأُنْسِيتُهَا . وَقَالَ الْحُمَيْدِيُّ عَنْ سُفْيَانَ قَالَ سُلَيْمَانُ لاَ أَدْرِي أَذَكَرَ سَعِيدٌ الثَّالِثَةَ فَنَسِيتُهَا أَوْ سَكَتَ عَنْهَا
Ibn ‘Abbas said that the Prophet (ﷺ) gave three recommendations saying “Expel the polytheists from Arabia, reward deputations as I did”. Ibn ‘Abbas said “He either did not mention the third or I have been caused to forget it. Al Humaidi said on the authority of Sufyan that Sulaiman said “I do not know whether Sa’id mentioned the third and I forgot or he himself did not mention it.”
A small note, I changed the above text from instructed
to recommended
as that is what the Arabic text translates to. This was a recommendation from the prophet and not an order, since we can see that not all the Calpihate Alsalhoon did it.
In the end, and in my opinion, based on the above evidence, I see no harm in people not of the book visiting any part of the middle east, but they are not allowed to commence Hajj rituals to the Kabba.