Salam, sometimes I hear people swear about Hell and I wanted to know if it would be kufr to stand around them? If I stood with them, would that be putting me beyond the pale of Islam? And what about hearing someone say the word Holy then swear? Would that be kufr for me to stand around them as well?
Being a disbeliever is not so easy term. Sometimes you may sound something that actually contradict with your belief system. But later you understand your fault. Then you should repent and promise to Allah not to do such act again.
You have to know that in Islam swearing on Allah is only allowed. Allah swears in the Quran with His several creations, it is Allah's authority.
I will say you that you should discuss your nearby mosque's Imam about the fact. If it is being your fault then make " Tawbah". And again if the fault is of your nearer people's ou had better avoid them.
May Allah bless and mercy upon you