I was wondering why no one has asked this question while the Hajj season is upon us.
I had the same exact question and wanted to know when the time is.
Here's my findings, please double check.
Firstly, if you have enough money to acquire gold that could have helped you make hajj then it's already time. A person is supposed to save with the more/same enthusiasm for hajj as he/she saves for their most beloved thing. Marriage/gold/diamonds.
Omar R.A used to order his companions to go around and find people who have not performed hajj and have enough money to perform one. When the list was made, he would order to collect Jizyah from them. Here's what Jizyah means for those who do not know.
Also, a person who has enough money and his wife is expecting or he has young kids should perform Hajj alone as he will be asked about himself first on the day of judgement as the wife and kids are exempt at that time but no the husband.
There are many more rulings but i think these two pretty much cover everything.
For more specifics consult local Alim/Mufti.
and Allah knows best.