If one fights a terrorist group that considers themselves to be Muslim and dies in the process is he considered as a martyr or is he sinning for fighting against Muslims?
Among the ways of becoming a martyr in Islam is being slain in the cause of Allah:
And if you are killed in the cause of Allah or die - then forgiveness from Allah and mercy are better than whatever they accumulate [in this world]. -- Qur'an 3:157; see also Qur'an 3:169 and Qur'an 47:4
... One who is slain in the way of Allah is a martyr; one who dies in the way of Allah, is a martyr; one who dies of plague is a martyr; one who dies of cholera is a martyr. ... -- Sahih Muslim 1915 a (sunnah.com)
and for self-defence:
He who is killed while protecting his property is a martyr, and he who is killed while defending his family, or his blood, or his religion is a martyr. -- Sunan Abi Dawud 4772 [grade: sahih] (sunnah.com); see also Islam Q&A
... if you should be killed [in the martial jihad] you will be among the living martyrs for whom God provides.
So maybe one of these applies to dying in battle against terrorists.
But it's complicated...
However, Islamic military jurisprudence is nuanced, and there's many different opinions. Killing anyone (Qur'an 5:32) and particularly Muslims (Qur'an 4:93) is ordinarily strictly forbidden. Sahih Muslim 96 a (sunnah.com) describes the Prophet's reaction after killing someone who said "There is no god but Allah". And terrorists may indeed be Muslims (although particularly ill-behaved ones).