Assalamu Alaikum,
I hope you find this helpful: http://islamqa.info/en/23339
Also, bear in mind that we do not please the creature to displease the Creator. Our testimony that "There is none worthy of worship except Allah and Muhammed is the last Messeger" is clear that our obligations to Allah comes first before the created.
Its also important to note that not acting upon knowledge we have is a grave offense in Islam as reported in this hadith:
"On the authority of Usamah ibn Zayd (ra) that he heard the Messenger of Allah (saw) sitting, "A man will be brought on the Day of Resurrection and thrown into the Fire. His intestines will be hanging out, spilling onto the ground, and he will go around like a donkey goes around a millstone. The inhabitants of the Fire will gather around him and say: "O so and so! What is your affair? Didn't you used to order us to do good deeds and forbid us from doing bad deeds?" He will reply, "Yes I used to order you to do good deeds, but I did not do them myself, and I used to forbid you from doing bad deeds, yet I used to do them myself". (al Bukhari, Engl. trans. vol.4, p.315, no. 489; Muslim and others)."
Though this applies mostly to scholars, it equally serves as a point of warning for us all.
It would be best if you found a scholar and explained the situation to him/her to know if expiation is upon you.
I leave you with this Hadith that has always helped in my states of confusion:
On the authority of Abu Muhammad al-Hasan bin Ali bin Abi Taib, the grandson of the Messenger of Allah, sallallahu ëalayhi wasallam, and who is dearest to him, radiyallahu ëanhuma, who said: I committed to memory from the Messenger of Allah, sallallahu ëalayhi wasallam, (the following words):
Leave that about which you are in doubt for that about which you are in no doubt.
[Al-Tirmidhi and al-Nasaíi related it, and al-Tirmidhi said: It is a good and genuine Hadith]
May Allah guide me and you, and protect us from our weakened heart.
Allah knows best.
