Is it allowed by Islam for an Islamic state to demand ransom from non-Muslims to return others taken as prisoners? Do they need to be formally at war?
Quran mentions about ransom in the context of war. The following is the verse
So when you meet those who disbelieve [in battle], strike [their] necks until, when you have inflicted slaughter upon them, then secure their bonds, and either [confer] favor afterwards or ransom [them] until the war lays down its burdens. Quran 47:4
As can be see in the verse, first option for captives of war is to free them by grace and second option being ransom for release of prisoners .The reason mentioned for ransom is for "war to lays down its burdens". This means Muslims are dragged onto a war by the disbelievers and it has cost Muslims wealth and lives due to this. So it is reasonable that prisoners are returned for ransom so that it pays for what costed the Muslims.
So to answer your Question, ransom is allowed for freeing prisoners of war. so yes, war is precondition for ransom.