Battle of Karbala's answer is great. There is an important difference he forgot to mention:
Jaber asked Imam Sadiq: Why don't you claim what belongs to you? (Why don't you take control of the Ummah, you're the rightful leaders?)
Imam Sadiq replied:
یا جابِر مَثَلُ الإِمامِ مَثَلُ الْکَعْبَةِ اِذْ یُؤتی وَلایَأْتی
Oh Jabir the position of the Imam (to his Ummah) is like the Ka'ba,
the Ummah should circle him, not him (The Ummah should go to him, not
the other way around)
Bihar al-Anwar Volume 35. Pg. 353
A prophet has a duty to walk around a guide people. The Imam's duty only becomes activated when enough people want him to be their leader. It's somewhat a democracy.
Even though Imam Ali was appointed by Allah he didn't oppose the other caliphes because people didn't want him.
For Imam Hussain the people sent letter for him to come and save them from Yazid son of Mu'awiah, letter they were threatened and decided not to back Imam Hussain. Then Imam Hussain was like well they don't want me anymore...let me go back to my own land... Yazid didn't even allow Imam Hussain returning to his homeland. Rest is history.
As you can see all other Imams, none of them had a governing role. They were simply there go guide whoever that comes to them.
(This is also a significant difference between Shia Imams and Sunni caliphes. For Sunnis basically anyone who's ruling is the caliphe. For Shias it's only who has divine authority.)
- Imam Ali was the Imam but it took him 25 years to become the caliphe.
- Imam Hassan was the Imam, but because Mua'wiyah gave money and land to the his followers he was left alone and had to come to peace with Mua'wiah.
- Imam Hussain was the Imam, but was brutally martyred.
- Imam Sajjad, Imam Baqir, Imam Sadiq were all Imams but just taught students about Islam which is why most Shia narrations are from them.
- Imam Kazim was Imam, but was imprisoned for many years. There were many days that he didn't even see the sun. The chains had softened his bones. He was still guiding others through his mediums
- Imam Reza was Imam, while he was positioned as Ma'mun's heir. During his time Ma'mun gathered many intellectual debates.
- Imam Jawad was an Imam even though he became an Imam at the age of 7
or 9. He also had intellectual debates.
- Imam Hadi and Imam Hassan al-Askar were on house arrest and distanced from people because the Abbasid government were controlling them and awaiting for the 12 Imam to be born. But still they were Imams. Only that they were communicating less frequently with their followers.
- Imam Mahdi became the Imam at the age of 5. For a period of time he was being communicated through his four deputies. Then his great occultation began.
Through out this period of 250 years...other than Imam Ali's few years it never it came a moment were people showed great desire for their Imam to take over and therefor they were never the ruler. People were more like: " Yeah we want you to be our Imam...but don't ask us to do anything too difficult."
An incident that wraps the whole idea was that group of people came to Imam Ali and wanted to pledge allegiance to him. He said come back tomorrow with shaved heads. Only 3 people showed up. ... So Imam Ali decided not to make any action, because the people didn't want him.