O you who have believed, upon you is [responsibility for] yourselves. Those who have gone astray will not harm you when you have been guided. To Allah is your return all together; then He will inform you of what you used to do. -- Qur'an 5:105
I feel I'm misunderstanding the sentence in bold. Several Islamic Prophets seem to have been harmed by those who have gone astray; as a few examples:
Prophet Yusuf was thrown down a well, was sold as a slave, and was imprisoned for many years.
Presumably, it's safe to consider that these Prophets were guided, yet harm came to them.
Question: If those who have gone astray will not harm the guided, why were many Prophets harmed?