Is 'web scraping' permitted in Islam?

Question

I am a web sites developer but I also do build web scrapers. The last time a question comes to my mind, is this kind of work triggers no issue in Islam? the sites to which I build scrapers contains no illegal content with regards to Islam rules(no nu womens, no alcohol buyers, no financial services that contains Riba ...). But the action of scraping Web sites content is what makes this question. especially that some sites consider this action as illegal. so the question is:

• what should I do if this considered as illegal in Islam?
• what should I do to the money that I gained from building such programs?
• actually I am working on a similar program, which is half the way already( I have been working for more than a month on it already) and a version of it is already deployed on a server and the client can use it? should I withdraw from it and keep it for the client, should I return the money to the client?

please give me responses, if you have already asked Chiekhs who know the subject really well. As I don't want answers that have no dalleel. Thank you.

Answer 1

Assuming you are scraping halah content. I think the hadith in At-Tirmdhi would help answer your question:

"... And the Muslims will be held to their conditions, except the conditions that make the lawful unlawful, or the unlawful lawful." At-Tirmdhi

This hadith talks about any conditions we have with others, and in your case, the rules that govern the internet.

Looking at this we have 4 categories:

1- Website made it permissible for you to scrap stuff - it is halal

2- Website didn't mention anything, But rules of internet allow it - it is halal

3- Website doesn't allow it - it is haram

4- Website allows it, But rules on internet do not. Then you will find many varying opinions on this whether it is halal (website allows it and website should not listen to internet regulators) or haram(internet doesn't). If this is the case, then it is recommended to stay on the safe side by choosing the haram opinion.

---

what do if it wasn't halal ? Obviously repenting to allah is the 1st thing you should do, then make a promise not to do it again.

What to do with the money? I couldn't find a direct answer for this question. Thus I recommend you ask a knowledgeable scholar who has some knowledge about monetary system. However, I found some questions which ask about haram money stolen from people, according to sheikh Ibn Othameien [ARABIC - YOUTUBE], you should return this money back to the original person. If you can't dins the original person or you stole from many people, then you should donate the money to the poor and needy on their behalf.

Answer 2

The basic rule of Islam is that by default all things are halal , unless explicitly made haram in the texts of Quran and Sunnah.

The case of scraping data is similar to how google crawls various pages and google itself is a massive data scraper. If the website is public , it means that the site is allowing access to its data to public. By scraping the publicly available data you are not stealing the data as the owner\website has not protected it nor does reading the data cause the data to be removed from the source. If the website allows sites like google to crawl over their pages and google itself uses this data for commercial benefits (maybe indirect) then I dont see any reason why another scraper would not be allowed.

If you are hacking in and getting protected information which is not to be disclosed to public then it is haram.

Moreover by scraping you are not even copying in its true sense as you are essentially recreating the data from publicly available information and the scrapped data becomes your own intellectual property. The scrapped data may be cleaner , have some transformations , different format hence it becomes the scrapers intellectual property(IP).

Apart from the web world, In the physical world also you get information from publicly available sources and generate your own work product which becomes your own IP.

Regarding terms of service then , the terms of service itself should be validated by the Islamic law to be applicable to a Muslim. Terms of service which disallow scraping data are not valid as the site allows public access to information hence this wont be considered as either copying data or stealing data in Islamic sense.

There are various lawsuits filed in this regards in the US courts by these websites against data scrapers and still there is a lot of uncertainty in this regard.

Allah knows best

Answer 3

As what I believe it is not haram, but if your intention of making this for bad stuff like exploiting glitches to hack into site to steal stuff then yes it is haram, you might want to make sure to check the robots.txt of site so the site will tell you where you can crawl in the site. The money you get from doing this as I said if your intention of making this program good, then the money is halal. You should keep building it if you think his intention for the program is going to help the world, but if you think that he would use it for bad stuff then you should return the money and stop continuing the application, you might need to check.