Verse 5:105 is not talking about physical harm as your question seems to point. Rather; it is talking about the responsibility of the believers: are believers responsible for their actions only? Or are they responsible for the actions of others? And should they refrain from enjoining the good and forbidding the evil or not?
Ibn 'Abbas and Abu Ja'far Al-Mansur both said about this verse that Allah says long as His servants obey Him in matters of halāl and harām, those who have gone astray [by not obeying Him in matters of halāl and harām] will not harm the believers as they bear no responsibility for others' actions.
Sa'id ibn Jubayr said the verse was revealed about the Jews and the Christians that however astray they have gone, the Muslims bear no responsibility for their actions, so Muslims should just take jizyah from them and leave them to their beliefs.
Az-Zamakhshari said that the believers used to feel very bad about how lost the disbelievers were at the time of the Prophet ﷺ and wish that they would embrace Islam. Allah told them that they should not overburden themselves psychologically (this is the harm the verse is referring to) if they were among the guided as they were not responsible for the disbelievers' actions.
It was not only the companions who had this feeling about the disbelievers, but also the Prophet ﷺ had similar feelings:
فَلَعَلَّكَ بَاخِعٌ نَّفْسَكَ عَلَىٰ آثَارِهِمْ إِن لَّمْ يُؤْمِنُوا بِهَٰذَا الْحَدِيثِ أَسَفًا
Then perhaps you would kill yourself through grief over them, [O Muhammad] if they do not believe in this message, [and] out of sorrow.
— Surat Al-Kahf 18:6
and similarly in Qur'an 26:3. However, there were companions that misunderstood verse 5:105, albeit in a different way from your question, that this verse meant they should not enjoin in good and forbid evil, that they should become passive and leave everyone to their beliefs and actions, and to think of religion as a relationship between a person and Allah that is a private matter and of no one else's concern (much like what is considered today to be the right thing to do by some Muslims and almost all non-Muslims). The hadith of Ash-Sha'bani tells us otherwise in the words of the Prophet ﷺ himself:
حَدَّثَنَا سَعِيدُ بْنُ يَعْقُوبَ الطَّالْقَانِيُّ، حَدَّثَنَا عَبْدُ اللَّهِ بْنُ الْمُبَارَكِ، أَخْبَرَنَا عُتْبَةُ بْنُ أَبِي حَكِيمٍ، حَدَّثَنَا عَمْرُو بْنُ جَارِيَةَ اللَّخْمِيُّ، عَنْ أَبِي أُمَيَّةَ الشَّعْبَانِيِّ، قَالَ أَتَيْتُ أَبَا ثَعْلَبَةَ الْخُشَنِيَّ فَقُلْتُ لَهُ كَيْفَ تَصْنَعُ فِي هَذِهِ الآيَةِ قَالَ أَيَّةُ آيَةٍ قُلْتُ قَوْلُهُ : ( يَا أَيُّهَا الَّذِينَ آمَنُوا عَلَيْكُمْ أَنْفُسَكُمْ لاَ يَضُرُّكُمْ مَنْ ضَلَّ إِذَا اهْتَدَيْتُمْ ) قَالَ أَمَا وَاللَّهِ لَقَدْ سَأَلْتَ عَنْهَا خَبِيرًا سَأَلْتُ عَنْهَا رَسُولَ اللَّهِ صلى الله عليه وسلم فَقَالَ " بَلِ ائْتَمِرُوا بِالْمَعْرُوفِ وَتَنَاهَوْا عَنِ الْمُنْكَرِ حَتَّى إِذَا رَأَيْتَ شُحًّا مُطَاعًا وَهَوًى مُتَّبَعًا وَدُنْيَا مُؤْثَرَةً وَإِعْجَابَ كُلِّ ذِي رَأْىٍ بِرَأْيِهِ فَعَلَيْكَ بِخَاصَّةِ نَفْسِكَ وَدَعِ الْعَوَامَّ فَإِنَّ مِنْ وَرَائِكُمْ أَيَّامًا الصَّبْرُ فِيهِنَّ مِثْلُ الْقَبْضِ عَلَى الْجَمْرِ لِلْعَامِلِ فِيهِنَّ مِثْلُ أَجْرِ خَمْسِينَ رَجُلاً يَعْمَلُونَ مِثْلَ عَمَلِكُمْ " . قَالَ عَبْدُ اللَّهِ بْنُ الْمُبَارَكِ وَزَادَنِي غَيْرُ عُتْبَةَ قِيلَ يَا رَسُولَ اللَّهِ أَجْرُ خَمْسِينَ رَجُلاً مِنَّا أَوْ مِنْهُمْ قَالَ " لاَ بَلْ أَجْرُ خَمْسِينَ مِنْكُمْ " . قَالَ أَبُو عِيسَى هَذَا حَدِيثٌ حَسَنٌ غَرِيبٌ .
Narrated Abu Umayah Ash-Sha'bani: "I went to Abu Tha'balah Al-Khushani and said to him: 'How do you deal with this Ayah?' He said: 'Which Ayah?' I said: 'Allah's saying: Take care of yourselves! If you follow the guidance no harm shall come to you (5:105).' He said: 'Well, by Allah! I asked one well-informed about it, I asked the Messenger of Allah (ﷺ) about it. [So] he said: "Rather, comply with (and order) the good, and stay away from (and prohibit) the evil, until you see avarice obeyed, desires followed, and the world preferred, and everyone is amazed by his view. Then you should be worried about yourself in particular, and worry of the common folk. Ahead of you are the days in which patience is like holding onto an ember, for the doer (of righteous deeds) during them is the like of the reward of fifty of those who do the like of what you do." 'Abdullah bin Al-Mubarak said: "It was added for me, by other than 'Utbah, that it was said: 'O Messenger of Allah! The reward of fifty men among us, or them?' He said: 'No! Rather the reward of fifty men among you.'"
— Jami' at-Tirmidhi, Vol. 5, Book 44, Hadith 3058
Al-Qurtubi narrates a story about Abu Bakr saying the same thing in a khutba one day. He said that people (these are people during the time of Abu Bakr) read this Verse 5:105 and bear on it a meaning that it is not meant to convey:
حَدَّثَنَا أَحْمَدُ بْنُ مَنِيعٍ، حَدَّثَنَا يَزِيدُ بْنُ هَارُونَ، أَخْبَرَنَا إِسْمَاعِيلُ بْنُ أَبِي خَالِدٍ، عَنْ قَيْسِ بْنِ أَبِي حَازِمٍ، عَنْ أَبِي بَكْرٍ الصِّدِّيقِ، أَنَّهُ قَالَ أَيُّهَا النَّاسُ إِنَّكُمْ تَقْرَءُونَ هَذِهِ الآيَةَ : (يَا أَيُّهَا الَّذِينَ آمَنُوا عَلَيْكُمْ أَنْفُسَكُمْ لاَ يَضُرُّكُمْ مَنْ ضَلَّ إِذَا اهْتَدَيْتُمْ ) وَإِنِّي سَمِعْتُ رَسُولَ اللَّهِ صلى الله عليه وسلم يَقُولُ " إِنَّ النَّاسَ إِذَا رَأَوُا الظَّالِمَ فَلَمْ يَأْخُذُوا عَلَى يَدَيْهِ أَوْشَكَ أَنْ يَعُمَّهُمُ اللَّهُ بِعِقَابٍ مِنْهُ " .
Abu Bakr As-Siddiq said: "O you people! You recite this Ayah: Take care of yourselves! If you follow the guidance no harm shall come to you. I indeed heard the Messenger of Allah (s.a.w) saying: 'When the people see the wrongdoer and they do not take him by the hand, then soon Allah shall envelop you in a punishment from him.'"
— Jami' at-Tirmidhi, Book 33, Hadith 11
