Awuthubilamihiminashaytangnirajim Bisimillahirahmanirahim, 


Dear brother, **The question need improvement.**


The critical point of the question is that *"more than the **agreed upon** labor cost"*. However, in the question there **is not a sentence state that the fair has been discussed ahead of time before the proceed of the raiding**. I did not see evidence you asked him "how much will it be?" at the very beginning. Otherwise, it could have never happened, since the price was not even requested at the time and a contract therefore, seemingly, to be established by omission of both parties. 
You also did not state when you asked him how much will it cost for him to wait.
"Regular pay" may not apply in this case, since the price was not defined at the very beginning.

Therefore, its should be discussed between you and the driver again (if possible), otherwise, you may need to consult with a knowledgeable Imam about this sticky situation.
May Allah makes you easier.