I am going to answer this question from another point of view. The most followed opinion that, yes magic is real, is already added as several answers in this question, so I thought of adding the other perspective instead, which indeed hasn't got much light and is far away from the majority view.
Defining our terms
When we speak about sihr, or magic, it is important that we define its meaning. Do we mean by it like a sorcerer casts a spell over another person, which makes him under his possession or do we have another understanding of it?
Now, the apparent meaning of the word «sihr» is magic.1 But if we go deeper, we will notice the Arabs used the word sihr (magic) in other contexts that really had nothing to do with our definition of magic.
As an example it is said in the Quran in Surat Al Mudathir, verse 24:
فَقَالَ إِنْ هَٰذَا إِلَّا سِحْرٌ يُؤْثَرُ
"All this is mere spellbinding eloquence handed down [from olden times]
Reading this verse in its context, we understand that a person from Quraysh was calling the Quran for sihr. And it is known that the prophet was called sahir in many places, due to the Quran. In tafsir Al Qurtubi, the word sihr here is explained to rather mean «deception» (khadi'a)2, not magic:
{ إِلاَّ سِحْرٌ يُؤْثَرُ }
أي يأثِره عن غيره. والسِّحر: الخديعة
The next verse (25) makes it clear that sihr is about words (in this context at least):
إِنْ هَٰذَا إِلَّا قَوْلُ الْبَشَرِ
This is not but the word of a human being."
Analyzing why these verses was reveled, it gives us more clues on the context of the word sihr. This is mentioned in the same tafsir of Qurtubi:2
(To give you some context, this is related to when the leaders of Quraysh was looking for a reason to warn people against the Prophet Mohammed, they tried calling him crazy, a liar, a soothsayer but they couldn't say that because they all agreed that it was a lie to call him that and they also knew the people would know they were lying, but because they wanted to stop him badly they had to come up with something, in order to not put the Prophet in a good light)
فقالت قريش للوليد: فما هو؟ ففكّر في نفسه، ثم نظر، ثم عبس، فقال: ما هو إلا ساحر! أما رأيتموه يفرق بين الرجل وأهله وولده ومواليه؟!
The (leaders) of Quraysh said to Walid, what is he then? (or what shall we call him?) So he thought, then he though then he frowned and said: he is nothing but a magician! Haven't you seen him (how he) separates between a man and his wife3,4 and his son and his loyals?!
So, they called him a magican, because (according to them) he, with "his" words (i.e the Quran), and message, somehow lead people to separate and divide. So they agreed that this was the best thing they should call him without lying. With other words, they warned people of listening to the message of the Prophet, because it would separate between people, i.e it was "magic" according to them.
This hadith5 also speaks about, albayan, or the ability to express oneself:
إِنَّ مِنَ الْبَيَانِ لَسِحْرًا
Some eloquent speech is as effective as magic.
Due to the vastness of Arabic, this is also possible to translate or interpret as:
Indeed, There is really something magically about the ability to really be able to express oneself.
Conclusion about the definition
With the analyze above we can conclude that the word sihr, could mean following things (depending on the context):
- Deceiving with ones speech, which may cause separation and other problems.
- Expressing oneself eloquently to convey a message (one could say rhetoric)
- Illusion (see explanation in section "Other verses")
Notice, I never added the apparent meaning which we already said means magic.
Is magic real?
These definitions and the contexts mentioned about will open ways of interpretations of certain verses in the Quran. We will start commenting on the famous verse 2:102 which tends to be the basic verse to support real magic.
And they followed [instead] what the devils had recited during the reign of Solomon. It was not Solomon who disbelieved, but the devils disbelieved, teaching people sihr and that which was revealed to the two angels at Babylon, Harut and Marut. But the two angels do not teach anyone unless they say, "We are a trial, so do not disbelieve." And [yet] they learn from them that by which they cause separation between a man and his wife. But they do not harm anyone through it except by permission of Allah . And the people learn what harms them and does not benefit them. But the Children of Israel certainly knew that whoever purchased it would not have in the Hereafter any share. And wretched is that for which they sold themselves, if they only knew.
This verse, is saying that sihr, but it isn't defining what sihr is anywhere except where it says:
cause separation between a man and his wife
And this description of sihr, likewise the one we found out when defining the terms, that the leaders of Quraysh called the Prophet a sahir, because he divides/separates between a man and his wife and so on, with his speech (message, Quran).
This, would then give us the possible conclusion that sihr - in this verse - really means that "deceitful speeches" was used - not magic - in order to cause married couple to separate.
If this is the case, because only this is mentioned in the verse, we shouldn't try to explain all other kind of magical things they might have learned.
P.S by reading Othello, you might come to know how some deceitful speeches really can act as magic and destroy a marriage (and lives).
Other verses
Surat Ta ha, verse 66, the verse about the "magicians"
قَالَ بَلْ أَلْقُوا ۖ فَإِذَا حِبَالُهُمْ وَعِصِيُّهُمْ يُخَيَّلُ إِلَيْهِ مِن سِحْرِهِمْ أَنَّهَا تَسْعَىٰ
He said, "Rather, you throw." And suddenly their ropes and staffs SEEMED to him from their magic that they were moving [like snakes].
In this verse, the so called magicians, clearly was throwing ropes and staffs, and the word "yokhayyalo", which means it was made to APPEAR, or made to seem (due to their "magic" or what we rather should call illusions), that they were moving. I.e it was made to appear by illusions that the ropes was moving like snakes. We also could come to the conclusion, due to other verses that the magicians planned a big show, which could be interpreted that they did in order to create and plan these illusions.
Now, we can also draw the conclusion that these called magicians never did any real magic, because of the wording "made to appear", i.e it was only an illusion. They prove it themselves when they in the same sura (ta ha) verse 69-70:
And throw what is in your right hand; it will swallow up what they have crafted. What they have crafted is but the trick of a magician, and the magician will not succeed wherever he is."
So the magicians fell down in prostration. They said, "We have believed in the Lord of Aaron and Moses."
If they really was magicians and really did magical stuff, they would not be shocked when Musa threw the staff, which then miraculously, by Gods permission became a real snake and swallowed their ropes. They would say, well I can do that too! But they knew that they were called magicians only due to their ILLUSIONS that they were able to create, and that they never could do anything as real as that miracle. Therefore they fell down in prostration and followed Moses.
This is also mentioned in Surat Al Araf, verse 116:
قَالَ أَلْقُوا ۖ فَلَمَّا أَلْقَوْا سَحَرُوا أَعْيُنَ النَّاسِ وَاسْتَرْهَبُوهُمْ وَجَاءُوا بِسِحْرٍ عَظِيمٍ
He said: Cast. So when they cast, they deceived the people's eyes and frightened them, and they produced a mighty enchantment.
saharo a'yona nas, which here is translated to decieved the people's eyes, i.e they created an illusion.
About Surat Al Falaq, verse 4:
وَمِن شَرِّ النَّفَّاثَاتِ فِي الْعُقَدِ
And from the evil of the blowers in knots
Zamakhshari comments about this in his tafsir:8
ولا تأثير لذلك
And it has no effects (i.e the evil blowing in knots) For more info read the tafsir8.
The hadiths
The famous hadith that the Prophet was «mashor» (made magic upon) is also denied by the one following this opinions because they feel it is contradicting the Quran on these verses:
The verses 17:47 and 25:8
إِذْ يَقُولُ الظَّالِمُونَ إِن تَتَّبِعُونَ إِلَّا رَجُلًا مَّسْحُورًا
when the wrongdoers say, "You follow not but a man affected by magic."
With other words, they think that it could not be true that he really was affected by magic, as the verse says: the wrongdoers (dhalimon) say that "you only follow a man affected by magic (mashor)"
The verse(s) after both chapters;
[25:9] انظُرْ كَيْفَ ضَرَبُوا لَكَ الْأَمْثَالَ فَضَلُّوا فَلَا يَسْتَطِيعُونَ سَبِيلًا
See what examples they have put forward for you. So they have gone astray, and never can they find a way.
Surat Duha, 93:3
مَا وَدَّعَكَ رَبُّكَ وَمَا قَلَىٰ
In no way has your Lord disregarded you, and in no way has He disfavored you.
The verse 5:67
وَاللَّهُ يَعْصِمُكَ مِنَ النَّاسِ
Allah will protect you from the people
This verse is also understand to contradict the hadith by the one holding this opinion. Therefore they do not accept it. I.e God would protect him, and not let him get affected by sihr.
Now, this question about the hadith being in Sahih Bukhari might be related to this: "Do Sunni Muslims consider whole Bukhari as 100% authentic?" and "Is Sahih al-Bukhari considered as 100% authentic by Sunni scholars?"
Which contains:
The total weak/fabricated hadiths in al-Bukharis collection comes to 52.
A recent scholar, Imam ibn Amin has written a well detailed book on the weak hadiths in Bukhari and Muslim. You can find it on the internet.
Not all 'Sunni-Muslims' believe Bukhari is 100% authentic
and
Well as a Muslim i'd like to say sahih al-Bukhari and sahih Muslim are the most authentic books of sunna but i think as they are human they could include some mistakes or fabrication.
Therefore, these scholars, chooses (interprets) these verses mentioned above as having the last word, and because the hadith is (thought to) contradict these verses, they reject the hadith. Because they also follow the opinion that hadith books could include mistakes and fabcrications, while the Quran doesn't. They also seem to have problems with the hadith, as they do not see how the Prophet, who is the best of mankind, would be affected by sihr (or a jinn performing the "sihr" on him), without being protected by God. They also have issues on how the comb, (which is a private item) containing his hair, was given to the one performing the sihr. Who gave the comb, or the hair from the comb to that man? The comb being a private item, should most likely be in his house. Did they somehow break in to his house? What about his wifes, who likely was home? Was it a jinn who took the comb? What about the "angels protecting" his house from satans? And that kind of sihr, would be counted as the worst kind of sihr possible to be done (by the one believing in that definition of magic). Would the best man, the prophet be affected by the worst kind of sihr? What about you being protected by God just by reading some verses and doing dhikr every day, wouldn't the prophet be the one most protected? Why wasn't this mentioned in the Quran? It seems to be a very important situation! So they got lots of issues with the hadith.
Some scholars that have denied the hadith that the prophet was affected by sihr (magic):
I could add other scholars alive (or tullab al-ilm), like Adnan Ibrahim, Tariq Suwadin, that reject this hadith, but the list would become rather long then.
Another hadith warning about practicing sihr13
This famous hadith which says:
"Avoid the seven great destructive sins ... practicing sihr ..."
Practicing sihr (or witchcraft) is considered one of the great destructive sins. But it doesn't imply that the sihr being practiced really works in that way people may expect. Even though it isn't working, the intention and the act done when practicing sihr is a great sin itself, not the result. Practicing sihr is connected to shirk, worshiping devils, jinns, murdering (which at least happen today), deceiving, foretelling (telling ghayb which only God knows) and more.
Three persons whose actions are not recorded
The hadith about persons whose actions are not recorded:15
The Messenger of Allah (ﷺ) said: There are three (persons) whose actions are not recorded: a sleeper till he awakes, an idiot till he is restored to reason, and a boy till he reaches puberty.
They also feel that, if someone has been affected by sihr, when for instance they do things that they cannot control, their actions shouldn't be recorded because they cannot control that, and that would be unjust. Because it is not mentioned in this hadith, they believe sihr cannot effect a person.
Good to know
- Most people who deny the reality of magic also deny the possibility
of the Jinn to posses humans. See "Which scholars deny possession
of humans by jinn?" for more information. It is also known, that
many, or most of the explanations of how sihr (magic) is done,
is with the help of a jinn. Removing the jinn from the sihr equation,
makes it harder to explain the "real magic".
- It is highly possible that magic (or rather sihr) was used by people
when talking about unknown things, which one really couldn't explain
that time, but truly some of the sihr is taught (as we see in verse
2:102). Therefore it is highly possible that if you turned on the
lights in a dark room, a person from that time might have interpreted
it as magic, if it wasn't explained well enough to him/her.
- Most people who talk about that sihr has been done to them or to someone they know, really doesn't base their statements on something convincing. At least all situations I've heard of are either possible to explain from a psychologically point of view or many times the story is very exaggerated (and the exaggeration could be explained in many ways).
Conclusion
There is much more one could add here, but in order to not make it too long I decide to make a conclusion.
With the analyze above, we can draw the conclusion that no "real magic" exists, or at least that there isn't something that forces us to believe in "real magic".
But most of the times it depends of what we mean by the word magic.
One scholar that follows this opinion, that I can come to think about now, is Tareq Al-Suwaidan. He discusses some of these points and many others, including ahadith and so on.