I’m confused as to who qualifies to be enslaved. People often say captives of war, but what exactly does that mean?
The opposite forces? Civilians in the town raided? How would you differentiate between a captive and non-combatant?
Also, when does the sultan enforce the jizya for the non-believer?, to my understanding Muslims cannot attack the ones who pay jizya, so let’s say the mujahideen attack a place, do they just keep collecting captives till the sultan tells them to stop?
The following are valid sources of obtaining slaves:
A Harbi can be captured and enslaved. These are then distributed as spoils of war and those who receive them can keep them as their own slaves or sell them to others. A Harbi is any disbeliever who is not a Dhimmi, Mu‘aahid or Musta’min - i.e. one who does not have any treaty with the Muslims. For details and references see Does islam allow killing or enslaving civilians in war? and Could Muslims purchase slaves which were kidnapped by non-Muslims?
A Dhimmi who breaks their treaty can be enslaved, because they become a Harbi by doing so. According to the majority of the madhabs this only occurs if the individual has directly broken the treaty, so a woman's treaty will not be voided just by of the actions of her husband. According to a minority the treaty of a person's women and children is automatically broken when he breaks it, and hence they can be enslaved for it. See Mawsoo’ah al-Fiqhiyyah 24/157.
A female apostate can be enslaved according to a very minority opinion. However, the majority only allow execution for her while most Hanafis imprison her until she either reverts or dies.
A person who is already a legally recognized slave (according to Islamic law) can be sold, gifted or inherited.
An offspring of a slave woman born through marriage (not concubinage) is a slave by default.
Enslaving a free Muslim is not valid, unless they were originally non-Muslims who only accepted Islam after capture. Enslaving Muslim rebels who fight against the ruler is also not valid. Slavery for theft or failure to repay a debt or punishment for any crime committed by a Muslim is forbidden. See Can Muslims be enslaved by other Muslims?
For a reference on the sources of slavery, see Mawsoo’ah al-Fiqhiyyah 3/299-301.
Jizyah can be enforced on the disbelievers in one of the following ways:
By treaty. Before attacking, the Muslims give an ultimatum to the disbelievers to either accept Islam, pay Jizya, or to be prepared to fight. If they accept either Islam or Jizyah then they can not be enslaved and are protected by a treaty. This applies to the men and also extends to their women and children and will continue to apply to their descendants who are born under Islamic rule, as long as the treaty is not violated by them.
وإذا لقيت عدوك من المشركين فادعهم إلى ثلاث خصال أو خلال فأيتهن ما أجابوك فاقبل منهم وكف عنهم ثم ادعهم إلى الإسلام ... فإن هم أبوا فسلهم الجزية فإن هم أجابوك فاقبل منهم وكف عنهم
When you meet your enemies who are polytheists, invite them to three courses of action. If they respond to any one of these, you also accept it and withhold yourself from doing them any harm. Invite them to (accept) Islam; ... If they refuse to accept Islam, demand from them the Jizya. If they agree to pay, accept it from them and hold off your hands.
— Muslim
Accepting Jizyah when the disbelievers agree to it is obligatory and there is no option to enslave them instead.
By military conquest. Meaning that the disbelievers do not surrender and are overcome through battle instead. Here the caliph has options on how to deal with the populace. Accepting Jizyah in this case is optional not obligatory and it is permitted to enslave* them instead. The details of the possible choices are mentioned in the next section.
*This means that it is categorically permissible to enslave them. There is no threshold after which enslavement 'must' stop and jizyah has to be taken instead.
The available choices depend on the type of prisoners:
Adult males: The ruler has the option to execute, enslave, ransom, exchange, free or impose Jizyah on them. Ransom and freeing are disagreed upon. Those who are enslaved will not pay Jizyah and those who pay Jizyah will not be enslaved.
Women and children: Execution is forbidden. Women and children also do not pay Jizyah, especially when it is imposed forcefully, see Is it obligatory for a non-Muslim woman to pay jizya?
As for ransom or freeing then there is disagreement among the madhabs: That is because women and children are counted among the spoils of war and the soldiers have a right to receive them, just like they have a right over the other movable property that was captured in the war.
Hence according to some madhabs, women and children are automatically enslaved on capture and it is obligatory on the ruler to distribute them. So it is not permissible to choose any other option unless the soldiers agree to waive their right.
Other madhabs allow the ruler to decide some other option unilaterally, as long as it serves some interest of the Muslims and he compensates their value to the soldiers.
For a reference on the choices regarding captive women and children see: Mawsoo’ah al-Fiqhiyyah 24/157-161
