In regard to the initial part of your question that you pointed to the story of Habu and Kabu (abel & cain) (هابیل و قابیل ), you also mentioned that " one side didn't fight back the murder but simply warn the other side of the hell fire"
So, the verse below (4:93) could be related with that story:
وَمَن يَقْتُلْ مُؤْمِنًا مُّتَعَمِّدًا فَجَزَاؤُهُ جَهَنَّمُ خَالِدًا
فِيهَا وَغَضِبَ اللَّـهُ عَلَيْهِ وَلَعَنَهُ وَأَعَدَّ لَهُ عَذَابًا
But whoever kills a believer intentionally - his recompense is Hell,
wherein he will abide eternally, and Allah has become angry with him
and has cursed him and has prepared for him a great punishment. (93)
or even we can pay attention to the following verse (Surah
al-Nisa:94) which could be related with the issue of Takfir
يَا أَيُّهَا الَّذِينَ آمَنُوا إِذَا ضَرَبْتُمْ فِي سَبِيلِ اللَّـهِ
فَتَبَيَّنُوا وَلَا تَقُولُوا لِمَنْ أَلْقَىٰ إِلَيْكُمُ السَّلَامَ
لَسْتَ مُؤْمِنًا تَبْتَغُونَ عَرَضَ الْحَيَاةِ الدُّنْيَا فَعِندَ
اللَّـهِ مَغَانِمُ كَثِيرَةٌ ۚ كَذَٰلِكَ كُنتُم مِّن قَبْلُ فَمَنَّ
اللَّـهُ عَلَيْكُمْ فَتَبَيَّنُوا ۚ إِنَّ اللَّـهَ كَانَ بِمَا
O you who have believed, when you go forth [to fight] in the cause of
Allah, investigate; and do not say to one who gives you [a greeting
of] peace "You are not a believer," aspiring for the goods of worldly
life; for with Allah are many acquisitions. You [yourselves] were like
that before; then Allah conferred His favor upon you, so investigate.
Indeed Allah is ever, with what you do, Acquainted. (94)
On the other hand, rationally (and even canonically) it is Wajib (mandatory) for us to defend ourselves. (of course by observing some points…). Even according to a Hadith from Imam Ali (a.s.):
(briefly): whoever of you who is able to defend his friend (against
the enemy), So he should defend him(his friend) as he defend himself.
احد من اخوانه فشلا، فليذب عن اخيه بفضل نجدته التي فضل بها عليه، کما
يذب عن نفسه، فلوشاء الله لجعله مثله. ان الموت طالب حثيت،...
The Hadith(tradition) above shows that as well as ourselves, it could be our duty to defend our friends too (of course by paying attention to its conditions according to the positions and so on)
Accordingly, even as you gave an example "if a Muslim is shooting at you or going to stab you, are one suppose to just let the other side do it? Or any type of self defensive actions are allowed in Islam in this type of scenario?"
Then this is your duty as an obligatory deed to save your life.
Or let me give you another example which is related with your question (but in a larger field):
In the war between Iran and Iraq (some years ago), Iraq attacked Iran. As you probably know, Iraq is an Islamic country, then shall we say Iran ought not defend itself because the soldiers of Iraq were Muslims? (and allowed them to kill themselves?)