As'salaamu alaikum akhi.
I've never heard of any prophet crying for years and years over a single transgression, but I HAVE read stories of Companions (R. A.) crying a lot over their sins, as well as some people recognized as Awliyallah (friends of Allah). I get where you're coming from. If Allah made forgiveness so "easy" to attain, why dwell on the sin?
Brother, people's reactions to their sins are a reflection of the quality of their relationship with Allah. I'm sure you've heard of the Darajat, right? We are not all on the same level in our Islam. We are not all on the same level when it comes to our Communion with Allah either.
For example, in Surat-ul-Anfaal (8), ayat 2, Allah says, "For the believers are those who, WHEN ALLAH IS MENTIONED, they feel a tremor in their hearts..." Honestly now, how often have YOU felt a tremor of fear (wajilah) just because someone MENTIONS Allah? The truth is that, in this ayat, Allah is describing people who are highly developed and and very sensitive in spiritual matters.
To a person like that, Allah is more than just a vague, abstract idea. His name is more than just a word that is heard and said. These people enjoy what is called "a near approach" to Allah. They are very much tuned in. It's deeper even than that. You see, they are deeply in love with Allah, and it is not a one-way transaction. Allah loves them back, and they FEEL the joy and pleasure of His love overwhelming them.
"Those of Faith are OVERFLOWING in their love for Allah..." Surat-ul-Baqarah (2), ayat 165.
"Therefore be patient with what they say, and celebrate the praises of your Lord before the rising of the sun, and before its setting; yes, celebrate them for part hours of the night and at the sides of the day, THAT YOU MAY HAVE PLEASURE AND JOY." Surat-ut-TaaHaa (20), ayat 130.
So, you have this highly developed person who has sinned against Allah. It doesn't affect him the way it affects you or me (and most of the world in our day and age). It affects him (or her) much more profoundly.
You see, you and I are pretty numb to spiritual matters. We are not tuned in (mentally and emotionally) to Allah as strongly as we could be and should be. We are in a constant state of distraction with YouTube, or the latest debate/argument on Facebook, or working, or SOMETHING else. If you live in America or Europe then you know sex, booze, drugs, and all sorts of craziness are all readily available to us, and -- let's keep it real, bro -- a lot of Muslims are indulging in all that stuff, just like the kufaar.
We sin all the time, and to us it's not a big deal. Allah forgives all sins, right? That's the attitude of a lot of people today. But a highly developed person? They don't want lose what they have with Allah. They feel really, deeply, truly sorry for their sins, and they feel anxiety over how it might negatively affect the sweetness of their Communion with their Lord.
"AND THOSE WHO FEAR THE DISPLEASURE OF THEIR LORD, FOR THEIR LORD'S DISPLEASURE IS THE OPPOSITE OF TRANQUILITY... " Surat-um-Ma'arij (70), ayah 27 - 28.
Insha'Allah, you got something out of this answer, brother.
Maa'as'salaam.
