Let me start with what I consider to be the only relevant difference between a Muslim and a Non-Muslim in this regard: when confronted with a part of Islam that contradict state of the art science, his ethical impulses and convictions, or his aesthetic intuitions, the Muslim will feel the need to reconcile such a conflict, however imperfectly, because he believes Islam to be of divine origin; the Non-Muslim will not feel such a need, because he is not convinced of Islam's claimed divine origin. This is the only difference: the Muslim is convinced that Islam has divine authority, the non-Muslim is not.
Apart from this, I am not aware of any reason why a Non-Muslim could not in principle reach a level of knowledge about Islam that would mean he would be recognized as being capable of performing ijtihad if he were Muslim. Learning the relevant disciplines is not a matter of being convinced that the results those disciplines produce are divine commandments and teachings, it is only a matter of acquiring a body of knowledge.
What fundamental rules would one use to understand (the message of) Islam?
To understand Islam as it is understood by (different sects of) Muslims, you would adopt two approaches that any Muslim could adopt if he tried to learn about Islam:
to understand what Muslims, lay and scholars alike, think with regard to specific questions, you would go about it with the methods of sociology; poll them, have interviews with them, have discourse with them, and make sure the atmosphere whenever you interact with them is such that they feel comfortable expressing their opinion without feeling the need to justify it to whoever they are talking to more than they feel the need to justify it to themselves
to understand Islamic theology, law, spirituality, etc., you would also do what a Muslim wanting to learn about those things does: you go see what the scholars say, what arguments they use, what level of disagreement they have, basically try to understand how closely they follow the relevant usul and how faithfully they evaluate the sources.
As for understanding the "true meaning" of Islam, this is an area where the question whether or not it really is of divine origin is relevant. If Islam is what it claims to be, then following the opinions of orthodox Islamic scholars is the way to go - by this i mean that following scholarly consensus seems the best available way to follow the intended meaning of the texts on those issues where there is consensus, and that following respected scholarly opinions on disputed issues will at least absolve one of potential mistakes in those opinions since they were arrived at by a faithful and well-intentioned deduction from the texts. If Islam is not what it claims to be, i.e. the Quran is not the literal word of an omniscient, omnipotent, unique God, then assessing it historically - as you would any other religion that is no longer believed to be true by anyone, such as ancient Greek religion around Zeus, ancient Norse paganism (although it is experiencing a revival), etc. - will lead to different conclusions. Of course this historical assessment would include "what did and do Muslims believe is the true meaning of Islam?"
In particular, such an evaluation would see no need to talk about Muhammad or his Companions reverently, or assuming that Muhammad's actions were not just justified, but the best example for mankind. It would see no need to look for expressions of God's will in the history of the caliphate or of theological discourse. In short, it would assess the development, spread, and expression of Islam throughout history with the same methodology and attitude as it would assess Hinduism, Socialism, or any other world view - where, when, and how did it start, who were its proponents, what were its teachings, what are its errors, how did it impact history, etc.
How is ones attitude towards scholars and their opinions? Could one agree with scholars opinions which surely are based by their fundamental rules, which might not be the same as yours?
I see no problem in approaching this as a Non-Muslim with the same attitude as an Islamic scholar would when it comes to questions about "what does Islam say on this or that matter". As I said in my second paragraph, one does not have to be convinced that a teaching or ruling is true or has authority in order to understand whether the teaching or ruling follows from the accepted sources via accepted methods.
How would one choose what is "Islamic" and what isn't? And what does one define "Islamic"?
If by "Islamic" you mean "acceptable part of the religion of Islam", then starting from the very fundamental doctrines - al-Ghazali says they are these: belief in Allah, belief that Muhammad was his Messenger and truthful in everything, and belief in yawm al-qiyamah - it seems it follows straightforwardly that if you accept these three points, and accept that the Quran really is the literal word of Allah (as the second of those three points implies), Muslim and Non-Muslim go about it the same way: find what the authorities (i.e. the scholars) say, and see which opinions survive in their discourse. To explain by example: if you want to know whether a claim is scientific, you check whether the experts of the relevant fields of science say about it, and what they think of the evidence presented in favor of the claim -- that's if you are not trained in the relevant field and methods, if you are trained then assessing it yourself and cross-checking with what other experts say is more immediate and provides a better picture.
If by "Islamic" you mean something other than "acceptable part of the religion of Islam", then please explain in a comment.
How would one avoid choosing opinions that increases ones reason of not believing in Islam as a religion from God?
An honest person, Muslim or Non-Muslim, would not approach the question of whether Islam's claims about itself are true with the bias of "I want to find this is true" or "I want to find this is false". An honest person evaluates evidence as he finds it. I'm not sure how to answer this point any other way than by rejecting the premise that a Non-Muslim is Non-Muslim because he is biased against Islam.
Could one base his understanding of Islam by the more salafi way of understanding it? i.e "well the salaf says this, and believes this... therefore this is Islam".
I'm not sure how to understand this point. All sects of Islam claim to base themselves on authoritative texts from the same era. Their difference is one of methods, not one of sources.
Perhaps you can explain in a comment whether this is a fair rephrasing of the point: "should legal and theological developments after the first few generations of Muslims be ignored as genuinely being Islam?" If that is your question, then the answer is that the Non-Muslim does not believe Islam to be of divine origin, and the question of "true Islam" is a matter of historical expression, not a matter of present imperatives. You will find that historically, Islam has changed and been expressed very differently in different times and places. One very specific question is that of the Quran being created or not created, where (I think) ibn Hanbal initially said "I don't know", and later fervently stood for "not created". Later developments in aqidah would discuss many issues where initially many opinions existed that would now be considered kufr.
With that in mind, the question of whether or not the surviving opinions of the salaf on one matter or another should dictate how we evaluate the legitimacy of modern Islamic discourse from the point of view of Islam itself is not one that poses itself to the Non-Muslim because he is not convinced that there is a divine intent behind Islam. What the Non-Muslim will do is look at the way in which Islam was expressed and understood over time.
If this did not address the specific question you had in mind with that bullet point, please address it in a comment.
How would one interpret matters that could be contradictory? Like if a hadith and a verse in the Quran contradict each other. Would one look at that in a more objective way? I.e. analysing it in-depth or just saying "I just found a contradiction!" and ignore other acceptable explanations? How would one know what true contradictions are (if there are any), and which aren't?
This is a quite broad question as I understand it and relates to what one would consider a convincing argument for or against a claim, basically: what epistemology would/should you adapt. My answer will be quite broad, but reflects my experience with how religious people (not just Muslims) argue for their religions.
This question seems to aim at whether or not the Non-Muslim is biased towards wanting to disprove Islam. As before, I'd like to point out that that is not what an honest person does - an honest person is also not biased towards ignoring the flaws of arguments that support their favored view.
Clearly, one has to see whether there really is something that is contradictory or not before concluding that that's the case based on superficialities. I'm not sure what you mean by "more objective way".
Issues like contradictions with reality or internal inconsistencies in the sources pose themselves for any religious texts, and numerous strategies for trying to mediate them have developed over time and are used to support the claims of any major religion. As with any argument, these issues boil down to a matter of how convincing something is. Due to its own doctrines, Islam faces some challenges that other religions do not face in the same manner:
- Islam claims that the Quran is the unaltered revealed truth from an omniscient God; this makes arguments like "the people who wrote it down just misinterpreted what they learned about God and wrote it down wrong" untenable in Islam.
- For the same reason, it is impossible to remain Muslim while saying about a single statement in the Quran that it is false; thus, even if all claims Islam makes were utterly convincing, except for a single statement in the Quran that is clearly false, then that single false statement would suffice as grounds for rejecting Islam - more than that, if one takes Islam's claims seriously, then a reasonable person has to consider any single clear error in the Quran as proof that the claims about Islam's divine origin are false.
- As with any religion that claims to have revelation from an omniscient God, one problem that Islam faces is that changes in its doctrine over time, changes in the proofs given for it after the death of Muhammad, etc., have a real problem of generating any power to convince. If God revealed truths to Muhammad, and finished the revelation in the Quran, then any arguments developed later in history seem more like ex post facto rationalizations than grounds for believing Islam. Phrased differently: for hundreds of years Muslims were content believing Islam without such arguments, then people started seeing problems with Islamic doctrine, then the arguments were developed. This is exactly the process you expect from a set of beliefs that humans made up and wanted to continue to adhere to. Based on that alone, even if those arguments seem convincing to some - as is the case with the arguments all religions present for themselves - the process by which the arguments formed contrasts starkly with the claims Islam makes about the Quran.
- Finally, there is always the option of rejecting something in Islam that you can reject without becoming a disbeliever, and remaining Muslim. For example, it is only fisq instead of kufr to reject a hadith that is not mutawatir. One such example would be the hadith about dipping a fly if it falls into your drink: one wing contains a disease, the other a cure, so push it into your drink, take it out, and drink. This is found in Bukhari and classified as sahih, yet it is not mutawatir. Rejecting that the hadith is true - any medical professional will tell you that what the hadith says is nonsense - would be fisq, not kufr. A similar case are the ahadith about the coccyx not decaying. You could remain Muslim while rejecting those ahadith. The much more relevant question is this though: would Islam still seem convincing to you if you clearly have to reject some of its sources based on what you know about reality?
After these initial thoughts, back to the question of your last bullet point: how do we evaluate arguments that are intended to resolve problems within Islamic doctrine.
- One would make sure to stick with the topic: always only examine the evidence with regard to whether a claim is true or false, not with regard to whether it would be nice if it was true, whether it is useful if people believe it. Ignore everything that may move hearts to support a claim, but says nothing about the truth of the claim.
- One would reject all arguments that make the claim unfalsifiable/uncriticizable - a claim that cannot even in principle be rejected based on any evidence is completely unconvincing. An example: take the claim that everything popped into existence 5 seconds ago with every person having consistent memories of a time before it, just the way we find in reality. By no means could you possibly show this to be false, and yet I assume everyone who reads this will immediately reject this claim - yet you reject it not because you can demonstrate it to be false, but because it somehow "breaks the rules" of convincing someone. The rule that it breaks is "being logically possible is not enough, you have to have some evidence to convince me". Thus, every argument that results in "no evidence is even thinkable that could disprove this argument" is completely unconvincing. Another such claim with regard astrology: the predictions of astrology work, but only for people who do not doubt it. It's impossible to scrutinize this claim because by doing so, you would already be doubting it.
- One would also reject every argument that simultaneously supports contradicting claims; such arguments usually have ambiguity at their source. For example, if someone were to argue that God really exists and is as Islam claims, but made it appear to people of all other religions that they have sound evidence for their religion, then that argument could just as well be applied to the ancient Greek gods: Zeus does rule the Pantheon, but made it look to people of all other religions that their religion is true. Whenever you see an argument in support of one claim A, ask yourself whether the same argument would work in support of any claim B that contradicts A. If that is the case, then the argument is completely unconvincing and not an argument at all.
- One would also reject arguments that say that the claim is valid, we just haven't found the relevant evidence yet. This may be considered a special instance of the previous category of bad arguments: it supports anything you want it to support. Examples: humans are actually from Mars? Yes, just haven't found the evidence yet. Dark matter consists of strawberry jelly? Yes, just haven't found the evidence yet. These arguments are not just unsupportive of a claim, they actually become arguments against the claim if the evidence you fail to find should be easily available. An example for this are claims of a world-wide flood that wiped out all humans except for a single family - the effects of such a flood would produce very reliable traces everywhere on Earth. Answering that objection by saying it was done miraculously is an instance of making the claim unfalsifiable.
This is not a comprehensive list of bad kinds of arguments presented in favor of many religions (not just Islam, but also Islam), but I will leave it here for now. If you have any questions regarding these, or any other ones, please point it out in a comment, or ping me in chat.
Since some people take the position that you either believe Islam or you don't, and it's simply a matter of faith whether you want to, let me address that as well. Such a position is not even an argument, it doesn't offer any opportunity to convince someone towards or against Islam. It is an acknowledgement that in the most important question about reality, instead of evaluating reasons for and against a claim, you follow your own inclinations, or what you have been raised with, or what you want to be true instead of that for which you have sound evidence of being true (and have solid arguments that whatever is being adduced as evidence against your position is false). Anyone who says similar things - that it's just a matter of faith, just choose whether or not you want to believe - about Zeus, or astrology, or alien abduction (all of these are less extravagant claims than those of Islam) would be considered irrational and as basing his beliefs on completely unconvincing grounds.
Now back to the question: if all the arguments to explain why the texts contain apparent contradictions suffer from flaws like the ones I mentioned, then it seems reasonable to me to say there really is a problem with the texts. As I said, the Non-Muslim sees no need to resolve any problems within the texts or between the texts and reality at all cost, and however imperfectly; the option of saying "I looked into this thoroughly, and the most convincing position really seems to be that the texts contain errors" is open to him.