The Quran restricts sex to four wives and with one's own slaves 4:3
If ye fear that ye shall not be able to deal justly with the orphans,
Marry women of your choice, Two or three or four; but if ye fear that
ye shall not be able to deal justly (with them), then only one, or (a
captive) that your right hands possess
The above verse says you can marry upto four (free) women but if you cannot treat them with justice (that is give them equal time) than marry only one. Or one can have what your "right hand posses", that means your own slaves. By this verse adultery outside marriage and other than slaves is clearly forbidden. But if one consider slaves as outside marriage, then yes sex outside marriage is allowed, but only with one's slaves.