The short answer is to take a local class on Islamic finance. It's a huge field of its own. Barring that, talk to a local scholar or imam you trust who can answer your specific questions.
As a software developer, I run into this all the time. "Should I build a software like X," where X could be woefully misused, abused, and misconstrued by users. Should I try to monitor every post, comment, and piece of content? Should I let people moderate each other?
One principle that comes to play widely here is the certainty principle. In a nutshell, how certain are you that the person will use your site/service/product for haram?
- If you are 100% sure, then it is haram to sell that product/service to that person.
- If you are 50-99% sure, then it is makrooh (disliked; rewarded if you avoid it) to do so.
- If you are 49% or less sure, then it is mubah (permissible; no reward/sin either way).
That's the fiqh of it. For the practical application, you need to be clever and think ahead.
Say you wanted to start a stock photography website. Digital photography, by itself, would be mubah (permissible) (so long as it remains digital). However, people can upload haram photographs.
What I would recommend is create a terms-of-usage. Something like:
Any content which is pornographic, overly sexualized, or illustrates alcohol or drug use, etc. at the discretion of the site, may be immediately removed and your account may be banned.
If you look at existing sites, you often see terms of use. This lowers the chance of someone using your site incorrectly by quite a lot. If they do, you have a legal and ethical way to evict their content, and possibly their account.
Which leaves you with a bigger problem: how do you check all the content? In short, you can't. Perhaps, wallahu a'lam, you have taken enough means, and whatever sneaks through is really someone else's problem anyway (they're uploading haram content to a service you worked hard to halalify).
Another tactic that works well is flagging/moderating. Allow users (who may be trustworthy or may not be) to flag or push out content which goes against the site terms.
Again, it's not a perfect system, but it doesn't need to be. These are some examples of principles you can apply and practical systems you can adopt to try to keep your business as squeaky-clean as possible.
In summary: I asked one of my teachers this question:
What do you say about a building a product or service that can be used by people to do unhalal stuff. For example if I create a tool for posting stories/books/writings online, and people use that platform to write Harry Potter type books, or sex books, or etc. We are definitely not building it for that (haram) purpose. But the issue is, they will come in our site and use it unethically, then what? Or a better question, is Mark Zuckerberg responsible for what happens on Facebook?
He replied:
In general, you are not responsible, nor is Mark Zuckerberg.
And Allah knows best.