The critical point of the question is that *"more than the **agreed upon** labor cost"* was demanded. However, in the question there **is not a single sentence stating that the fare had been discussed ahead of time beforehand**. I did not see evidence you asked him "how much will it be?" at the very beginning; it could very well have never happened.

If the price was not even requested at the time and no contract thereby established, then where is the "agreed upon" labor cost?

You also did not state when you asked him how much will it cost for him to wait: "Regular pay" may not even apply in this case.

It is unfair to hold the driver to any contract that he never transacted by binding him to a labor cost that he never agreed to, even if he (or other drivers) had agreed to a similar price in the past.  Therefore, it should be discussed between you and the driver again (if possible).