It is a symbol of unity (all participants in the prayer facing the same direction)
As stated in the Quran:
For every nation there is a direction to which they face (in their prayers). So hasten towards all that is good. Wheresoever you may be, God will bring you together (on the Day of Resurrection). Truly, God is Able to do all things. And from wheresoever you start forth (for prayers), turn your face in the direction of Al-Masjid-al-Haram (at Mecca), that is indeed the truth from your Lord. And God is not unaware of what you do. (Quran 2:148-149)
Also The qibla was not always oriented towards Mecca. The first Muslims prayed towards the al Aqsa Mosque in Jerusalem. Around sixteen months after Prophet Muhammad and his followers migrated from Mecca to the city of Medina, the qibla was changed to the Kaba. According to accounts by Prophet Muhammad's companions, the change happened very suddenly. During the noon prayer, Prophet Muhammad, may God praise him, received a revelation from God instructing him to, "Turn your face towards the Masjid al Haram".
“Thus, we have made you real believers in Islamic Monotheism, true followers of Prophet Muhammad and his legal ways, a just nation, witnesses over mankind and the Messenger a witness over you. And We made the Qibla (prayer direction towards Jerusalem) which you used to face, only to test those who followed the Messenger from those who would turn on their heels (i.e. disobey the Messenger). Indeed, it was great (heavy) except for those whom God guided. And God would never make your prayers to be lost (i.e. your prayers offered towards Jerusalem). Truly, God is full of kindness, the Most Merciful towards humankind.”
“Verily! We have seen the turning of your (Muhammad) face towards the heaven. Surely, We shall turn you to a Qibla (prayer direction) that shall please you, so turn your face in the direction of Al-Masjid- al-Haram (at Mecca). And wheresoever you people are, turn your faces (in prayer) in that direction...” (Quran 2:143-144)
Thus changing the direction of prayer establishes Mecca as the fixed central point for worship. It establishes a common sense or purpose.
It is important the Muslims make every effort to face the right direction when praying; however, slight deviations do not invalidate a person’s prayer. Prophet Muhammad said, What is between the east and the west is qibla
Hope your doubt is cleared.