To explain this we need to perform a tafsir or go through the whole story. The best help i could find is tafsir at-tahriri wa-tanwir of Sheikh ibn Ashour (May Allah have mercy with him) and al-Kashaf of Imam az-Zamakhshari.
First of all note that this story is an example given by Allah.
This is the summary in a few words:
The Believer is the one who apparently has less than the other in almost anything which would be appreciated in this life (dunya) while the non-Believer has two gardens, many children and wealth, so the garden or wealth destroyed later in the story is equal to his deeds or belongings which in the afterlife wouldn't serve or help him in any means, while the good deeds of the -in this life poor- believer would serve him and save him in the aftermath.
So this story has taught us to thank Allah for what he offers or has given us, as stated in the hadith.
A second lesson is the importance of advise and the importance to seek advice of good people and the possible result of not doing so.
A 3rd lesson is that nothing is forever in this life.
And maybe an other lesson is the fact that even if non-Believers might have given wealth and many other things a Believer can make dua' and this way has "a direct connection to Allah".... and many more.
Note directly at the end of this story Allah quotes in a few words the example of the life of this world.
On the other hand i recommend you to read mindfully the characteristics of the owner (ظَالِمٌ لِنَفْسِه) of the gardens and the characteristics of the gardens (لَمْ تَظْلِمْ مِنْهُ شَيْئاً) if you are able to read/understand Arabic.
The View of Imam az-Zamakhsahri
One possible explanation for the transformation from dual plural to singular is given as follows: Each of us may have two gardens: One in this life with all the good things in it and an other in the aftermath, but we may plant in this life what we may get in the aftermath, so maybe the garden we then receive might not be as beautiful as the garden we have had in this life, because of our deeds, faith etc. This is more or less a conclusion of az-Zamakhsaris explantion: as he didn't explain the gardens and their products as only being: grapes and fruits, but also gold and silver so he described garden as this life's wealth in general. So according to az-Zamahkhari the non-Believer wouldn't have any share of the gardens which are promised for the believers.
The classical tafsir view
As you said in the two first verses the Quran speaks of two gardens. That should be clear as this verses describe the wealth of the non-Believer.
Now the first occasion the Quran quotes only one Garden is (18:35): this is logical as one can't enter two gardens instead of entering one of them: So in:
And he entered his garden ودخل جنته,
...
I do not think that this will perish - ever ما أظن أن تبيد هذه أبدا,
the verse is referring to only one garden.
Now the problem starts with (18:36), where the readings of 'Asim (hafs/shoaba), Abu 'Amr, al-Kissa-i, Hamzah, Khalaf and Yaqoob used the singular going ahead in the meaning of the Verse before while the other riwayat and qiraat confirm the dual plural by saying
ولئن رددت إلى ربي لأجدن خيرا منهما منقلبا
instead of
ولئن رددت إلى ربي لأجدن خيرا منها منقلبا (Kufi/Basri-Qiraa'a: i.e: Hafs)
which refers to two gardens, but the meaning
I will surely find better than this as a return
applies for both readings. So one may understand it as follows: the Kufi quraa' and Abu 'Amr and Yaqoob from Basra go ahead in the meaning of the non-Believer being in one of his gardens and reflecting about it, while the other quraa' recall the fact that the man has two gardens, so anything good which applies to one should apply to the other!
Az-Zamakhashri explained the dual plural form here again, as one transient/passing garden and one remaining/resident garden.
What is said about (18:35) applies for (18:39),
Again (18:40-41) are still quoting only one garden and this time we can't refer to any accepted qiraa'a to explain the possible problem! So one possible interpretation is that the Believer makes a which or dua' to Allah to destroy one of the gardens, if this is true, then any later problem is solved. If not we still have to explain why the Quran doesn't say anything about the 2nd garden: And this could be a solution from the Arabic language where in cases singular stands for plural and vice versa. But i couldn't find a specific tafsir claiming this for these verses.
But scholars (like ibn Kathir) here differ and go on the line of az-Zamakhari and say that
"It may be that my Lord will give me [something] better than your garden"
refers to the garden in paradise! Ibn Ashor here suggests an other option: that this is a dua' the believer made for himself and against the non-Believer. While
"and will send upon it a calamity from the sky, and it will become a smooth, dusty ground," refers to the garden of the believer in this life.
Note that verse (18:42) implies that all the fruits have been destroyed.
Conclusion
So we are at the start: Assuming we are talking about two physical gardens in this life: Maybe both gardens have been destroyed (if the linguistic rule of singular is replaced by plural and vice versa applies or if we assume the verse 18:42 to include the final statement), maybe only one (if that rule doesn't apply).
If we assume one garden refers to the good things in this life and the other to those of the aftermath: Than of course the non-Believer might have destroyed his garden in Paradise or aftermath only.
And Allah knows best!