It is a symbol of unity (all participants in the prayer facing the same direction) as well as symbol of respect for the Ka'aba. In the following Ayah, Allah makes it very clear that where we face during prayer isn't really that important.
Righteousness is not that you turn your faces toward the east or the west, but [true] righteousness is [in] one who believes in Allah , the Last Day, the angels, the Book, and the prophets and gives wealth, in spite of love for it, to relatives, orphans, the needy, the traveler, those who ask [for help], and for freeing slaves; [and who] establishes prayer and gives zakah; [those who] fulfill their promise when they promise; and [those who] are patient in poverty and hardship and during battle. Those are the ones who have been true, and it is those who are the righteous.
