I am not a scholar, just saying. And forgive me for the long answer as the question is philosophically complex.
Islam forces the principle "the end doesn't justify the means." So inaction would be the appropriate course of action in this scenario and pulling the switch would be a great sin.
Islam establishes rules similar to 'Categorical Moral Imperatives' but with a hierarchy and also permits a level of utilitarianism within the code. An action is divinely judged according to intention, not the action. And good and bad isn't inherent but determined by God.
I'll split the answer into two parts. (1) why in Islam the end doesn't justify the means and (2) application of the principle to the trolley dilemma and other scenarios.
(1) Anas ibn Malik reported: The Messenger of Allah, peace and blessings be upon him, said that if the end of the world is right in front of your eyes while you have in your hand a sapling, then you should plant it.
It doesn't make much sense if one doesn't believe in God. Rationally, the end is what matters and the means have no intrinsic value. As a Muslim, however, we believe in a mix of fate and freewill and that the meaning of life is to worship God and do what he commands. It is fated for you to face this big problem. You didn't choose or tell these workers to come on the track. God is the one who put you in this situation and he is testing you. He tests whether you will submit to his rules and his will, or will you persist or follow your ignorant or harmful whims. But You are also endowed with freewill, and you choose how to react. The end is not in your hands. Whether you choose to do good or bad, God already knows what you will do, and based on infinite factors, God chose these specific workers to die on this day by the train if you chose to do nothing or chose that that specific one worker to die on this day by your choice to sacrifice him. God has decreed that he will make all humans die. So if these workers were to die today, God chose that they die today. God simply chose that they die this way, as a tribulation for some people, as a mercy for some, and as a punishment for some.
Regardless, you as the person holding the switch don't know anything. You don't know who God decreed to die. You could do everything in your power to kill someone, but God saves him. You could do everything in your power to save someone, but God kills him. You don't know why this situation is taking place and the result or the benefit of the result. God is the ultimate utilitarian (if that's an appropriate thing to say). Whatever happens, it is the best thing that could ever happen.
As the person holding the switch, the only thing that matters for you is what you choose to do. The end is not in your hands, but you get rewarded or punished for your choice. You are still held responsible by God and you earn what you intend. You decide based on two factors. First, you need to determine the Islamic ruling of the means. An action can be prohibited, dislike, permissible, liked or obliged. If the means is prohibited by God, then there is no legitimate means and there is nothing you can do. The end is not in your hands regardless of what you do anyway. You are not responsible and God is decreed this to happen; and God is most wise and merciful. You sin and might get punished if you commit what's prohibited. But God also took your sin into account, so your parents for instance can take comfort in that the result was still calculated by God. But You will still get punished for your intention because you have freewill to choose. You have no intrinsic power, God was the one who enforced your intention.
On the other hand, if the means is an obliged by God, then it is an obligation that you have to fulfill. You get reward for sure if you do it, and you might get punished if you don't do it. If the means is disliked, permissible or liked, you as a human based on what the physical and social laws that God established in our universe, you assume and act according to what you think would result in the best outcome. If you don't act, you don't get punished and if you act, you get rewarded. So, in this case, You can think in utilitarian terms. But still keeping in mind that there is a possibility that you could be wrong. And you must keep in mind that it is not because you acted that the good outcome happened. The fire doesn't burn if you touch it and water doesn't quench your thirst if you drink it. It is God who burns and quenches thirst. It is laws that he established and which he enforces. So if you want to quench your thirst, you intend to drink, so God gives you the power and moves you to the kitchen and give you the ability to drink. He fulfills your intention. And quenches your thirst based on a law he established: water = no thirst. But God could sometimes break his laws, he is not bound by them. So you keep in mind that you are just doing what you think is result in the best thing. And your intention and endeavor is rewarded regardless of the outcome. In this case, you think based on the end, but it isn't because the end matters in and of itself but because God ordained that you try to achieve the best ends with his legitimate means.
This is not saying that believers should do good deeds just because of the reward. The reward and warning is methods used by God to motivate people. A believer should do it for God, because he appreciates and worships him. God also insists that for someone to truly believe, he must want what God also wants. For instance, you should give charity to others because, for one, God commanded you to do so and you obey, but also, you do it because you want to help them as much as God wants to. Again, God of course wants greater good for others that anyone and he does what he wants regardless of you help or not. You don’t help someone who needs; God will just get someone else to do it before you get even born. And again, you are still responsible for your intentions. If you gave charity to showoff, you will not be rewarded. If you give kill to save, God might forgive you. And God is aware of what you keep in your soul.
I am not saying, however, that atheists don’t care about the means. We believe also in human nature that God established in humans. The human regardless of his conviction will act as a human. We all have a moral compass programmed into us.
I personally think, with no proof other than the little Vsauce video, that most people in the trolley situation would actually not kill the one person. When confronted with the decision of killing someone, their human nature would make them submit and surrender to inaction. The best they could is look for a third option.
And I think, after the accident, the ones who did nothing would be disturbed for a short time. They would know that they couldn’t do anything, they would know it was an accident and they wouldn’t be legally responsible. On the other hand, people who chose to kill the one would be disturbed significantly. They would feel great remorse in that they actually caused someone to die. They would think ‘maybe I shouldn’t have done that’. The event will probably chase them for a very long time in their life.
I hope this clarifies why "the end doesn't justify the means."
(2) Application of the reasoning to the trolley dilemma and other scenarios: Islam strictly prohibits killing in any case, even killing yourself. There are only two exceptions: capital punishment for murder ordained by court and self-defense.
God says “Because of that, We decreed upon the Children of Israel that whoever kills a soul unless for a soul or for corruption [done] in the land (capital punishment) “it is as if he had slain mankind entirely. And whoever saves one“it is as if he had saved mankind entirely. And our messengers had certainly come to them with clear proofs. Then indeed many of them, [even] after that, throughout the land, were transgressors.” God therefore not only warns people of great punishment for killing, but also insists that in case of illegitimate means, we shouldn’t think in utilitarian terms because killing one is liking killing all humanity. But also, God motivates people to save others using the other legitimate means by indicating a huge reward.
In our case, since the only means to save the five people is pulling the switch and killing the one, then there is no legitimate means. Therefore, you shouldn’t act. But if there is any other possible legitimate way, you should do everything you can because saving one is like saving the entire humanity.
The rule thus applies in all similar scenarios. You can’t push someone fat off a bridge to make him fall to block the train and save the five people. A doctor can’t kill a healthy patient to take his organs and save five other sick patients. The military can’t blow up a plane full of people to stop it from crashing into a building. Passengers on a lost ship can’t kill sick people and eat them for the majority to survive. Passengers on a ship can’t push someone people off to stop it from sinking and saving the majority. You can’t nuke a city to stop a country from fighting you (it doesn’t count as self-defense because the citizens are unarmed, innocent and aren’t pointing guns at you). You can’t sacrifice civilians as collateral damage to bomb terrorists.
A professor in Harvard argued that since cars kill many people, it should also be an illegitimate means by this logic. But movement and transportation, by your legs, by a camel or horse, or by a car or a plane, is already established by God as permissible. So in this case, you should think in utilitarian terms. There should be regulations to reduce harm or a total ban depending on the different places and time.
And only God knows best. I could be wrong.
Despite all this, I need to note that God is extremely merciful and we shouldn't judge. Someone who chooses to kill the one to save the 5 could be forgiven easily. God cares about intentions and he takes into account all various factors. God is just but more merciful. To put this into perspective, here's several sayings by the God and the prophet:
Quran:
God accept the repentance of those who do evil in ignorance and repent soon afterwards; to them will God turn in mercy: For God is full of knowledge and wisdom (4:17)
Despair not of the Mercy of God, for God forgives all sins: for He is Oft-Forgiving, Most Merciful. (39:53)
Hadith:
Allah possesses 100 mercies, only one of which He has sent down to the world, through -which His creatures are merciful to each other. And 99 mercies has He held back, and with them He shall have mercy on people on the Day of Judgment.
When Allah decreed the creation, He wrote in His Book which is with Him on His Throne: My mercy prevails over My wrath(justice).
Once after a battle, there was a woman among whow was running then suddenly she found a boy among the captives, she took him and she held him close to her chest and she nursed him. The Messenger of Allah said then, “Do you think this woman will throw her child in the Fire.” The companions said, “No, as long as she can prevent it.” Then the messenger said, “I assuredly swear that Allah is more merciful toward His servants than that woman with her son.”
The Messenger of Allah (ﷺ) said, "Allah, the Exalted, has said: 'O son of Adam! I shall go on forgiving you so long as you pray to Me and aspire for My forgiveness whatever may be your sins. O son of Adam! I do not care even if your sins should pile up to the sky and should you beg pardon of Me, I would forgive you. O son of Adam! If you come to Me with an earthful of sins and meet Me, not associating anything with Me in worship, I will certainly grant you as much pardon as will fill the earth."'
Prophet of Allah (ﷺ) said "There was a man from among a nation before you who killed ninety-nine people and then made an inquiry about the most learned person on the earth. He was directed to a monk. He came to him and told him that he had killed ninety-nine people and asked him if there was any chance for his repentance to be accepted. He replied in the negative and the man killed him also completing one hundred. He then asked about the most learned man in the earth. He was directed to a scholar. He told him that he had killed one hundred people and asked him if there was any chance for his repentance to be accepted. He replied in the affirmative and asked, 'Who stands between you and repentance? Go to such and such land; there (you will find) people devoted to prayer and worship of Allah, join them in worship, and do not come back to your land because it is an evil place.' So he went away and hardly had he covered half the distance when death overtook him; and there was a dispute between the angels of mercy and the angels of torment. The angels of mercy pleaded, 'This man has come with a repenting heart to Allah,' and the angels of punishment argued, 'He never did a virtuous deed in his life.' Then there appeared another angel in the form of a human being and the contending angels agreed to make him arbiter between them. He said, 'Measure the distance between the two lands. He will be considered belonging to the land to which he is nearer.' They measured and found him closer to the land (land of piety) where he intended to go, and so the angels of mercy collected his soul".
So I can't just say that someone who killed will go to hell or something. He could be forgiven. I am just saying that this is a wrong choice, to extent of my knowledge, so don't do it in the future.