I wish to know, who as an entity can be at war with another party according to Islam? A state, an instititution, a family or a person? Also, what reasons and opponents are permitted to declare war or to be at war?
Whoever that is responsible or supports the wrongdoer. Meaning if one person wrongfully kills another Muslim, then he is to be brought to court. If others stop this then they should should be punished, though I'm not sure of the extent of the punishment of others vs the individual murderer. Likely if one supports another's doing then they are deemed as equal. But don't quote me on that.
Fight in the way of Allah those who fight you but do not transgress. Indeed. Allah does not like transgressors. https://quran.com/2/190
Say a person from country X comes and kills 1 person from country Y, and the country Y kills him and then kills 10 others and names it as collateral damage. That is NOT acceptable. It is a transgression.
[Fighting in] the sacred month is for [aggression committed in] the sacred month, and for [all] violations is legal retribution. So whoever has assaulted you, then assault him in the same way that he has assaulted you. And fear Allah and know that Allah is with those who fear Him. https://quran.com/2/194
Basically the act and its punishment should be proportionate.
What reasons to declare war?
Allah does not forbid you from those who do not fight you because of religion and do not expel you from your homes - from being righteous toward them and acting justly toward them. Indeed, Allah loves those who act justly.
So if one entity attempts to remove us from our homeland that we are to fight back.
Or as always self-defense is accepted.
Having that said the first approach is to forgive or sign a peace treaty. If the enemies don't stop then the Muslims most protect themselves. Under no circumstance Muslims can enter the lands of their enemies. <-- such doing can only be instructed by God or his messengers