From Tafsir al-Maududi:
فَلَمَّا رَأَىٰ أَيْدِيَهُمْ لَا تَصِلُ إِلَيْهِ نَكِرَهُمْ وَأَوْجَسَ مِنْهُمْ خِيفَةً قَالُوا لَا تَخَفْ إِنَّا أُرْسِلْنَا إِلَىٰ قَوْمِ لُوطٍ
But when he saw that they did not stretch out their hands towards it, he had a mistrust about them and felt a fear of them in his heart. They said, "Don't have any fear: for we have been sent to the people of Lot."
Quran 11:70
Some commentators are of the view that Prophet Abraham was afraid of them because, from their refusal to eat his meal, he had concluded that they had come to him with some evil intention. For, according to the Arab tradition, if some strangers refused hospitality, it meant that they had not come as guests but as enemies to plunder. But this view is not supported by the succeeding verse.
The wording of their answer shows that Prophet Abraham had understood from their refusal to eat that they were angels. But he was afraid of their visit because he knew that the coming of the angels in the human shape did not bode well: So he was fearful that they might have come to take them to task for some offense committed by the people of his household or of his habitation or by himself. Had it been the same thing that the commentators presumed, then their answer should have been, "Don't fear us for we are angels from your Lord. " But the answer they gave to remove his fear, "Don't have any fear for we have been sent to the people of Lot (and not to you)," shows that Abraham knew that they were angels but was troubled in his mind as to the reason of their visit in the human shape. This was because the angels came in human shape when some one was either to be put to a hard trial or punished severely for some offense.
According to Maududi, Ibrahim had feared that since the visit of angels in human form does not bode well, and that they might of come for some wrongdoing that his family committed. He knew that they were angels when they refused to eat.
Ibn Kathir says:
(But when he saw their hands went not towards it (the meal), he mistrusted them,) This means that he felt estranged from them.
﴿وَأَوْجَسَ مِنْهُمْ خِيفَةً﴾
(and conceived a fear of them.) This is because angels are not concerned with food. They do not desire it, nor do they eat it. Therefore, when Ibrahim saw them reject the food that he had brought them, without tasting any of it at all, he felt a mistrust of them.
﴿وَأَوْجَسَ مِنْهُمْ خِيفَةً﴾
(and conceived a fear of them. )
﴿فَلَمَّا رَأَى أَيْدِيَهُمْ لاَ تَصِلُ إِلَيْهِ نَكِرَهُمْ﴾
(But when he saw their hands went not towards it (the meal), he mistrusted them,) When Ibrahim saw that they were not eating, he became scared and frightened by them.
Ibn Kathir says that when the angels refused to eat or taste any of the food, Ibrahim felt estranged from them and feared for what they had come for but then was relieved when the angels said they were sent to the people of Lut.