Within a primarily Muslim country, if a non-Muslim commits a punishable sin, would they be punished in accordance to Islamic Law?
This may sound paradoxical, but the Islamic shari'ah permits other faith communities to be governed by their own shari'ah. There is freedom of religion in this sense - people are free to practise their faith, including religious law. Additionally, Islamic obligations such as zakat or military service that do apply to Muslims are not applied to "people of the covenant."
We have numerous examples of this from the time of the Prophet (saws). When the Prophet (saws) got to Madinah he established a constitution (saheefah) for the citizens. After basic statements establishing the rights of citizens, it was permitted for leaders of faith communities to settle disputes within their communities according to their laws, as long as it didn't contradict the earlier clauses pertaining to overall societal harmony.
For example, when the community of Jews in Madinah were deciding how to punish a man who had committed adultery and consulted the Prophet (saws), the Prophet (saws) asked for a copy of the Torah and asked for the relevant passage to be read to him. And we know this about personal law from many other sources as well. In history, the various Islamic empires had systems wherein different faith communities could adjudicate according to their faith and traditions. Other examples including alcohol consumption and raising and consuming pigs. Non-Muslims are permitted to do this in Islamic states. They are permitted to have their own courts for civil law (marriage, inheritance, divorce, etc.) and for infractions considered a sin in their own religions, Muslim jurists would rule by that - but not when it was a sin in Islam (e.g. drinking alcohol or eating pork) (see here). This kind of tolerance is exemplary and noteworthy for all those who claim that Islam is oppressive or intolerant.
My answer and proofs so far are restricted to 1) People of the Book and 2) personal law or within-community infractions. It doesn't answer the question of what happens when a crime is committed outside the community, and if the community is not Ahl al-Kitab.
For the first question, there is an incident from the khilafah of 'Umar b. Abd al-Aziz, commonly known as the fifth righteous khalifa in Islam. He was shocked once at hearing of incestuous practices in the Zoroastrian community (within Islamic political dominion). He wrote to al-Hasan al-Basri asking for advice - and he was told that once the people of the covenant paid the jizyah, they are to be left alone to live as they please without interference from the government. (I heard this from my teacher, and it is referenced also in Maududi's book "The Rights Of The People of Covenant In The Islamic State.")
I don't know the answer to the second missing thing above (what happens when the infraction is not limited to one community - what kind of law applies then).
In Darul Islam, non-Muslims (actually Dhimmi people) generally have same social responsibilities and rights as Muslims, and they are applied to same laws. For example, either a Muslim kills a Dhimmi or a Dhimmi kills a Muslim, the same law is applied. However, there are differences in marriage and heritage. For example, there is no heritage between a Muslim and a Dhimmi.
Moreover, a Dhimmi has to respect to Islamic prohibitions although they are not prohibited in their religion. So, they can't do explicitly whatever is against Islamic laws. They can't ring their bells explicitly and can't speak out their beliefs. Their temples, drinks, and pigs are not interfered unless Dhimmi people do what they do explicitly. If a muslim damages to these drinks, pigs.. he is obliged to compensate the damage.
Besides, Dhimmi people are excluded from specific duties assigned to Muslims. For example, they can't enter Harem Area in Mecca. There different views in details.
I had compiled the answer from web sources which I trusted. Now, here are some published sources for my answer. el-Ahkâmü's Sultaniye is the most well-known and classic book of public law according to its introduction. The book is written by Imam Maverdi (born in 974). Ölçüler (published in 2006) is a book for Hanafi Fıqh. I make one citation from Ölçüler to complement citations from el-Ahkâmü's Sultaniye.
Which law is applied?
According to Ebu Hanifa, if a Muslim deliberately kills a non-Muslim, Qısas is applied and the killer Muslim is killed. But according to Imam Shafi'i, Imam Malik, Imam Leys and Imam Ahmad the killer Muslim isn't not killed. (Ölçüler, p. 492) I couldn't find a summarizing sentence in el-Ahkâmü's Sultaniye but I infer from its the two pages (el-Ahkâmü's Sultaniye, p. 433) that killer non-Muslim is killed.
If Dhimmi people have a conflict about daily jobs, they can go to their judges. They are not intervened about that. But if they apply to Muslim judges, the judge make a decision according to Islamic laws and apply necessary punishment. (el-Ahkâmü's Sultaniye, p. 277)
Punishments of zina is also discussed in el-Ahkâmü's Sultaniye (p. 419). According to Shafi'i, there is no difference between Muslim and non-Muslim in punishment, but according to Ebu Hanafi, there are some differences.
Limitations for non-Muslims
There are two kinds of requirements for non-Muslims to be counted as Dhimmi: requirements to be Mustahak and requirements to be Mustahab (to maintain agreement).
Non-Muslims have to fulfil Type I requirements whether the requirements are stated in the agreement or not. If they violate, it means they break the agreement. Here are the six requirements about being Mustahak:
- Not saying anything bad about Qur'an and not claiming that Qur'an is distorted.
- Not accusing The Prophet may peace and blessings be upon him as a liar and not insulting him.
- Not claiming that İslam is false and not criticizing it.
- Not tending to a Muslim woman either for zina or marriage, and not enforcing them.
- Not causing fitna in Islam by pretending to be a Muslim, not dissuading Muslims from Islam, not invading Muslims' properties and religion.
- Not being on the enemy side and not financing them.
Non-Muslims don't have to fulfil Type II requirements if they are not stated in the agreement. If they are stated in the agreement, violating them doesn't means breaking the agreement. Violaters are enforced to fulfil. Here are the six requirements about being Mustahab:
- Wearing white and wearing Zunnar to show their status.
- Not building buildings higher than those of Muslims.
- Not making heard by Muslims their bells, reading their books, the words about that Uzeyr and Mesih are sons of Allah.
- Not showing their crosses or idols, pigs, action of drinking.
- Not burrying their deads explicitly and lamenting loudly.
- Not riding racehorses.
(el-Ahkâmü's Sultaniye, p. 275)
If the agreement is broken, non-Muslims move to whereever they trust. (el-Ahkâmü's Sultaniye, p. 277)
There is also a citation about doing things explicitly:
If a person keeps drink bottles beside himself or in his shop, and he is a Muslim, the drink bottles are poured and he is punished. If he is Dhimmi, he is only punished for keeping drinks explicitly. According to Ebu Hanafi, drink bottles of a Muslim is not poured because non-Muslims can drink it. (el-Ahkâmü's Sultaniye, p.467)
You can search for other issues which I didn't gave a proof. There are different views on details as I said, because there are not many ayahs or hadiths about details.