Contradictory to what you say, Ka'bah is not a mere building, it is a house of worship. Rather, it is the first ever place of worship the mankind was allotted.
Indeed, the first House [of worship] established for mankind was that at Makkah - blessed and a guidance for the worlds. Qur'an 3:96.
Most importantly, Allah has called it "My House" in the following Ayah and maybe a few more Ayaah.
And [mention, O Muhammad], when We designated for Abraham the site of the House, [saying], "Do not associate anything with Me and purify My House for those who perform Tawaf and those who stand [in prayer] and those who bow and prostrate. Qur'an 22:26
Though Allah does not dwell in that house, nor does he need a house, he has still called it as "My House" (Allah's house or Baythullah). This signifies its importance and sacredness. And we have no right to call it a mere building.
The most important significance of Ka'bah is to unify the people.. If there was no Qibla, muslims would pray facing anywhere and they would consider any place they personally respect as their Qibla. Like the muslims are now divided on other bases, they would have been divided based on Qibla.
After realising its importance, if a person still finds no harm in just pointing the feet toward the Ka'bah it should be acceptable because there is no hadith or Ayah supporting that it is Haraam or makrooh. When Qur'an and Hadith don't support or contradict an act, we are left with Ijmaa'. When we consider Ijmaa', some scholars like Mufti Ibraheem Desai find it disrespectful, some don't. It's up to your heart now. If a person's heart is very rational and finds no harm, its up to him. But, if his heart considers it disrespectful, and he still points it purposely, he is a transgressor.
What is significance of Kaabah besides a place for doing worship?
- It unites people toward one Allah whether black or white, slaves or masters, rich or poor, young or old.
- It signifies equality in the eyes of Allah as all face toward one Qibla no matter where a person hails from, what his caste or profession or income is.
- Further reading: Significance of the Ka'bah.
Are we supposed to love and respect the building?
When Allah has called it "My House" despite of the fact that he doesn't need a house, it is clear that he has stressed its importance. If you love and respect Allah, you must love and respect what he emphasizes.
Will that give us sawab?
Loving what Allah emphasizes is a consequence of loving Allah Himself. Allah loves who love Him unconditionally.
When Allah loves you, the whole heaven loves you:
Narrated Abu Huraira:
The Prophet said, "If Allah loves a person, He calls Gabriel saying: 'Allah loves so and so; O Gabriel, love him.' Gabriel would love him, and then Gabriel would make an announcement among the residents of the Heaven, 'Allah loves so-and-so, therefore, you should love him also.' So, all the residents of the Heavens would love him and then he is granted the pleasure of the people of the earth." Sahih Al-Bukhari Volume 8: Book 73: Number 66.
ۚ وَاللَّهُ عَلِيمٌ حَكِيمٌ (And Allah is the All-Knower, the Wise)
اللهُمَّ علِّمنا مَا يَنْفَعُنا وَانْفَعْنا بِما عَلَّمتَنَا وزِدْنَا عِلما