I'm writing a book/essay about solving mathematical problems. I write about praying and Allah replying, is this haram? I know for sure that writing about seeing him, or depicting him is wrong but I'm not sure about him replying. Is it mockery to make it like a dream? I've looked it up but can't find an answer.
In general, we can talk about answers we receive in prayer. However, this has to be true at least to our own understanding and consciousness. Telling an invented story about a response is abuse and haram because it is a misuse of the name of Allah. There may be a cause to write about our personal experience in prayer but a publication on math problems is for sure not supposed to include this.
First of all, I don't see the connection between 'solving mathematical problems' and 'praying'. It's a non-sequitur: it doesn't follow. And nor do I see the point of writing something merely for someone not to follow. That's generally not the point of writing.
Any depiction of Allah, as you point out is haram. This includes turning Him into a figment of your imagination or dream. It is also haram to compare yourself to Moses or any other prophet.