Lets say there are villages in settlements outside of dar al islam. Are muslim armies allowed to kill and enslave them by surprise if they are in dar al harb? Is asking them to convert or pay jizya a pre-requisite before attacking them? Are civilians of a settlement, tribe or village killed if they are non-combatants?
If dawah to Islam or Jizyah has reached them in the past then it is allowed to attack them by surprise. It is not necessary to renew the warning before attacking as otherwise they will make preparations and hence remove the strategic advantage from the Muslims. Especially places near the border are considered by scholars as those where dawah has already reached. We know of a number of examples from the seerah where the people were attacked by surprise without a renewal of dawah:
واقعدوا لهم كل مرصد
sit in wait for them at every place of ambush
كان رسول الله صلى الله عليه وسلم إذا غزا قوما لم يغر حتى يصبح، فإن سمع أذانا أمسك، وإن لم يسمع أذانا أغار بعد ما يصبح
Whenever Allah's Messenger (ﷺ) attacked some people, he would never attack them till it was dawn. If he heard the Adhan (i.e. call for prayer) he would stop, and if he did not hear the Adhan, he would attack them immediately after dawn.
كتبت إلى نافع فكتب إلى أن النبي صلى الله عليه وسلم أغار على بني المصطلق وهم غارون
Nafi' wrote in reply to my letter that the Prophet (ﷺ) had suddenly attacked Bani Mustaliq without warning while they were heedless ... Nafi' said that Ibn 'Umar had told him the above narration and that Ibn 'Umar was in that army.
Some fiqh references:
يجب إن لم تبلغهم الدعوة ولا يجب إن بلغتهم لكن يستحب
It is obligatory to give the intimation if dawah has not reached them. And it is not obligatory to give intimation if dawah has previously reached them, but it is recommended.
قال أحمد: إن الدعوة قد بلغت وانتشرت، ولكن إن جاز أن يكون قوم خلف الروم وخلف الترك على هذه الصفة لم يجز قتالهم قبل الدعوة ...وقال مالك: أما من قارب الدروب فالدعوة مطروحة لعلمهم بما يدعون إليه
Imam Ahmad ibn Hanbal said: The invitation has already been conveyed. However if there are any people settled behind the Romans and Turks who have not received the invitation then it is not permitted to fight them before giving invitation ...
Imam Malik said: As for those who are close to us, invitation has already been conveyed to them. As it is in their knowledge what the Muslims will invite them to do.
Killing during battle:
During the attack it is by default permissible to kill any adult male Kafir. That is because the default rule about all disbelievers is that they can be lawfully killed.
Being a 'civilian' or 'combatant' is not a factor rather kufr and having the ability to fight is the factor. So Muslims are spared because they are not Kafir. And women, children are spared because they can not fight due to physical limitations. While most adult males can be killed as they have the physical and legal ability to fight.
فاقتلوا المشركين حيث وجدتموهم ... فإن تابوا وأقاموا الصلاة وآتوا الزكاة فخلوا سبيلهم
Kill the polytheists wherever you find them ... But if they should repent, establish prayer, and give zakah, let them [go] on their way.
أمرت أن أقاتل الناس حتى يشهدوا أن لا إله إلا الله وأن محمدا رسول الله ... فإذا فعلوا ذلك عصموا مني دماءهم وأموالهم
I have been ordered (by Allah) to fight against the people until they testify that none has the right to be worshiped but Allah and that Muhammad is Allah's Messenger ... so if they perform that, then they save their lives and property from me.
Some fiqh references:
بأن الله تبارك وتعالى حرم دم المؤمن وماله إلا بواحدة ألزمه إياها، وأباح دم الكافر وماله إلا بأن يؤدي الجزية أو يستأمن إلى مدة
Allah has made the blood and property of a Muslim sacred, except when he himself makes a violation which renders it lawful. And Allah has made the blood and property of a disbeliever lawful except when he pays Jizya or is given asylum for a limited period of time
إذا أسر الرجل منهم في القتال أو غير القتال مثل أن تلقيه السفينة إلينا أو يضل الطريق أو يؤخذ بحيلة فإنه يفعل فيه الإمام الأصلح من قتله أو استعباده أو المن عليه أو مفاداته بمال أو نفس عند أكثر الفقهاء كما دل عليه الكتاب والسنة. وإن كان من الفقهاء من يرى المن عليه ومفاداته منسوخا
If a male is made prisoner in fighting or without fighting, such as if he is cast adrift from a ship, or loses his way (and ends up in Islamic territory), or is taken by some ruse then the Imam will deal with him according to what best suits the Muslim interests. The choices being between executing him, or enslaving him, or freeing him as a favor, or exchanging him for ransom or for life (prisoner exchange) according to most of the jurists as this is evident from the Quran and Sunnah. And some of the jurists held that freeing as a favor or exchange are abrogated.
والأصل فيه أن كل من كان من أهل القتال يحل قتله، سواء قاتل أو لم يقاتل
The rule is that it is permissible to kill every one who is capable of fighting, regardless of whether they practically fight or do not fight.
And rationally all able bodied men are treated as reserve forces which can be conscripted when needed. And they themselves might at any time freely decide to participate in fighting. The choice to personally fight or enlist in a professional army is not the deciding factor in Islamic law. Only those who are explicitly protected can not be killed.
It is not permitted to kill some categories like women and children, and some madhabs add additional categories like old men and reclusive monks, based on analogy or based on other ahadith with debated authenticity and interpretation.
فنهى رسول الله صلى الله عليه وسلم عن قتل النساء والصبيان
Allah's Messenger (ﷺ) forbade the killing of women and children.
However there are some situations where even the killing of women and children is permissible:
If they participate in fighting it is permitted to kill them. Similarly if they contribute to fighting indirectly then it is permitted to kill them. This includes for example when they give advice, incite others or are obeyed by the fighters (such as queens or kings) etc.
وقتل من نسائهم امرأة واحدة، كانت قد طرحت الرحا على خلاد بن سويد فقتلته، فقتلت لأجل ذلك
And one woman (of the Banu Quraiza) was killed because she had killed Khallad bin Suwaid by dropping a grinding stone upon him.
ومن قاتل من هؤلاء أو النساء أو المشايخ أو الرهبان في المعركة قتل لا نعلم فيه خلافا وبهذا
And whoever fights in a battle from among the women or old men or monks can be killed. We do not know of any disagreement on this.
If the weapon or strategy is indiscriminate then it is permissible to use it if there is no other feasible way to overcome the enemy without using it. The Muslims will not intentionally aim for the women and children but their killing is forgiven when it happens by accident or inadvertently. Examples of this include sieges where women and children may be harmed because of the lack of supplies or spread of diseases, or raids at night when it is hard to discriminate who is who because of darkness and proximity, or area of effect weapons like causing floods by breaking dams, or raining arrows, or bombardment, or setting fires to settlements etc. where it is not possible to control who gets hit.
and besiege them
وسئل عن أهل الدار يبيتون من المشركين، فيصاب من نسائهم وذراريهم قال هم منهم
The Prophet (ﷺ) ... was asked whether it was permissible to attack the pagan warriors at night with the probability of exposing their women and children to danger.
The Prophet (ﷺ) replied, "They (i.e. women and children) are from them (i.e. pagans)."
نصب النبي صلى الله عليه وسلم المنجنيق على أهل الطائف
The Prophet ﷺ used a catapult against the people of Taif
ويجوز حصار الكفار في البلاد والحصون والقلاع، وإرسال الماء عليهم، ورميهم بنار ومنجنيق وما في معنى ذلك من هدم بيوتهم، وقطع الماء عنهم، وإلقاء حيات أو عقارب عليهم ولو كان فيهم نساء وصبيان
And it is permissible to besiege the disbelievers in settlements, fortresses and castles. And to let lose flood water on them, and to shoot fire and catapults at them, and whatever else comes under the meaning of that such as demolishing their homes, and cutting off their water, and throwing snakes and scorpions on them even if there are women and children among them
After the attack is successful and the Muslims have gained control over the disbelievers, they can be imprisoned. And what can be done with prisoners is the following:
They can be killed if the Muslims think that is in the Muslims' interest. And anyone who can not be killed in battle can also not be killed after imprisonment. While anyone who can be killed during the battle can generally also be killed after it. However a prisoner who accepts Islam after capture can not be killed, even if he participated in fighting beforehand. Also some madhabs have noted the exception that children fighters can not be killed if they are captured alive.
They can also be enslaved instead. And this option applies especially to the women and children. Women and children are not combatants, except for rare occasions, yet they can be enslaved.
وأقر قريظة ومن عليهم، حتى حاربت قريظة، فقتل رجالهم وقسم نساءهم وأولادهم وأموالهم بين المسلمين
(The Prophet) allowed Bani Quraiza to remain at their places till they fought against the Prophet (ﷺ) again. He then killed their men and distributed their women, children and property among the Muslims
فأما النكاية التي هي الاستعباد فهي جائزة بطريق الإجماع في جميع أنواع المشركين، أعني: ذكرانهم وإناثهم وشيوخهم وصبيانهم صغارهم وكبارهم إلا الرهبان
Enslavement is permitted by way of consensus for all categories of the polytheists, their men and women, old and young, and the common people and the elite with the exception of monks.
Or they can be exchanged for money or for Muslim prisoners. Or Jizyah can be forcefully imposed on them. Or they can be freed as a favor and returned to the disbelievers. There is some difference among the madhabs on the permissibly of these choices.
فإذا لقيتم الذين كفروا فضرب الرقاب حتى إذا أثخنتموهم فشدوا الوثاق فإما منا بعد وإما فداء حتى تضع الحرب أوزارها
So when you meet those who disbelieve strike [their] necks until, when you have inflicted slaughter upon them, then secure their bonds, and either [confer] favor afterwards or ransom [them] until the war lays down its burdens.
The choice is to be made based on what suits the interest of the Muslims, it is for example blameworthy to free them when the enemy is still strong and numerous and the prisoners will rejoin their ranks and strengthen them against the Muslims:
ما كان لنبي أن يكون له أسرى حتى يثخن في الأرض
It is not for a prophet to have captives until he inflicts a massacre in the land
Again, being a combatant is not necessary for capture and for these options to be exercised. As evident in the following hadith where a travelling disbeliever was captured and ransomed without his personal involvement in fighting:
كانت ثقيف حلفاء لبني عقيل فأسرت ثقيف رجلين من أصحاب رسول الله صلى الله عليه وسلم وأسر أصحاب رسول الله صلى الله عليه وسلم رجلا من بني عقيل ... فقال بم أخذتني ... فقال ... أخذتك بجريرة حلفائك ثقيف ... ففدي بالرجلين
The tribe of Thaqif were allies of Banu 'Uqail, and Thaqif captured two of the Companions of the Messenger of Allah ﷺ. And the Companions of the Messenger of Allah ﷺ captured a man from Banu 'Uqail.
Thereupon he (the prisoner) said: Why have you taken me as prisoner? ...
He (Muhammad ﷺ) said: 'I captured you because of the wrongdoing of your allies Thaqif.' ...
He was then ransomed for two the two (muslim) men (who had been taken prisoner by Thaqif).
It is a misconception among some that all non Muslims lands should be attacked. Muslims should only sent missionaries to spread the message of Islam to non Muslim region. That is how Islam spread to countries like India in the first century, not through military movement.
Muslims can attack a place to liberate a place if the ruler is cruel and the people there are being oppressed
And what is [the matter] with you that you fight not in the cause of Allāh and [for] the oppressed among men, women, and children who say, "Our Lord, take us out of this city of oppressive people and appoint for us from Yourself a protector and appoint for us from Yourself a helper"? Quran 4:75
Other than this, there is no point in starting war on peacefully living people.
Whenever they kindle the fire of war, Allah puts it out. And they strive to spread corruption in the land. And Allah does not like corruptors. Quran 5:64
In modern times, we can see starting war is easy but to stop it is very difficult. All the people including civillian in both sides suffer, all their daily life gets effected by war. So no point in arguing for war, when there is no clear instructions in Quran or Sunna to start war on all non muslim nations. Actually this concept of dar-al-islam(house of Islam) and dar-al-harb(house of war) to denote all non muslim regions, is not from Quran or Sunnah, it is concept created by later scholars.