The structure of answer is: First, I try to establish the reliability of Naskh and to prove that it is doesn't really contradict "Infallibility of Allah" rather it is the wisdom of Allah. Next, I provide explanation for the quoted hadith and "alleged" verse.
Most Sunni Muslims strongly believe in Naskh (Abrogation). The principle of Naskh is mentioned in Qur'an and it is not something made up by scholars.
We do not abrogate a verse or cause it to be forgotten except that We
bring forth [one] better than it or similar to it. Do you not know
that Allah is over all things competent? [Surat Al-Baqarah]
This verse was explained by lot of classical scholars. One example would be:
Ibn Jarir Al-Tabari (d. 310 H.) said:
“Allah means with (Such of Our revelations as we abrogate): we
abrogate the ruling of a verse by another to change and replace this
ruling. This is by making Halal to become Haram and vice versa or to
let what is permissible to become impermissible. Thus, abrogation is
only concerned with matters of commands and rulings such like:
permissibility and impermissibility, prohibition and Halal … etc.
Therefore, previous nation’s news and stories have no abrogation…”
[Tafsir Ibn jarir (jami’e Al-Bayan):1/521]
There are explanations from Ibn Kathir, Al Qurtubi, Mujahid, Qutada and more, which you can find here.
There are many hadiths as well which provide examples of Abrogation. You can find examples of abrogations as well in the above link.
Wisdom behind Naskh
Naskh doesn't contradict "Infallibility of Allah". It is important to realize that, just because an ayah was revealed and later recanted, does not mean that Allah made a mistake. Rather, this is wisdom of Allah. Allah reveals the first verse while knowing he is going to abrogate it with a second and there is a wisdom behind it.
When Qur'an was being revealed to Arabs, it was being revealed in stages and it was something new for them and it was supposed to be implemented in their lives and not just recite it. So, these changes were brought gradually and not suddenly, and this is a mercy of Allah so that people could adjust to these new prescriptions.
An example would be that of verses in Qur'an about drinking wine. Wine drinking was very common in the pre-islamic Arabia and although, it was evil, it was considered with high esteem (and that is even now). So, there were three verses which gradually lead to prohibition of intoxicating substances. It was revealed in stages - Qur'an 4:43, Qur'an 2: 219, Qur'an 5:93-94.
According to Dr. Abu Ameenah Bilal Philips, there are some wisdom behind abrogations-
First, as I told you before, for the ease of implementation of a prohibition or any other ruling among people.
Second, it is a test for human beings in which they are closely required to follow certain laws in certain situations while in others they are not required. It tests the willingness to submit and it is a test of steadfastness upon the religion. And that is why the Ansar and the Muhajir were the best of all ummah after the Prophets (peace be upon all of them). The Ansar and the Muhajir never doubted what was revealed to them. When there was a verse that came to them, they would immediately implement them.
Third, it shows the love of Allah for the Islamic Ummah. When a naskh repeals one law with one more difficult, Allah gives the believers an opportunity to earn more reward with their obedience. Greater the difficulty, greater the reward.
I recommend reading this book for further knowledge on Naskh — Ulum Al-Qur'an : An Introduction to the Sciences of The Qur'an by Ahmad von denffer (Beautiful Book!)
Coming to the "alleged" verse quoted and regarding the hadith quoted in the question:
Imam ibn Hajar Al Asqalani says in his commentary on Saheeh Bukhari:
Umar said: "When this verse came down I approached the Prophet peace
be upon him so I asked him: Should I write it down?' It is as if he
hated that" Then Umar said: "Cant you see that if the old man if he
commits adultery he does not get the whip, and that if the young man
if he commits adultery he gets stoned?" (Ibn Hajar Al Asqalani, Fathul
Bari, Kitab: Al Hudood, Bab: Al I'tiraaf bil Zina, Commentary on
Hadith no. 6327)
We can clearly see that the Prophet (salallahu alayhi wassalam) never wanted the verse to be written down as a part of Qur'an, and it was never meant to be part of it. And we do know that Prophet prohibited writing down things at one time and encouraged at others. This is because he wanted to prevent people from taking down everything he says (he wanted to differentiate between what was part of revelation and his commands (More)) as part of Qur'an. Scholars of Islam are unanimous that the recitation of this verse has been abrogated but its ruling still remains in effect.
However, the reason why Umar (radiallahu anhu) got worked up and wanted to include that in Qur'an because he was afraid that people would argue about the punishment of stoning and would cancel it ( And how true Umar was when we see people arguing about stoning!). However, the companions did not allow him to do so because they all knew it was abrogated. Umar himself knew it but he got emotional because he feared for people of future (i.e. us).
Imam ibn Hajar Al Asqalani has in his commentary
In the verse whose recitation has been abrogated but its ruling
remained, and it has happened what Umar feared. A tribe from the
Khawarij or most of them and some of the Mu'tazilites rejected the
stoning. And it was reported by Abd al Razzaq and Al Tabari from
another view that Ibn Abbas said that Umar said "There will come a
people that will lie (or disbelieve) in the stoning" (Ibn Hajar Al
Asqalani, Fathul Bari, Kitab: Al Hudood, Bab: Rajam Al Hublah min Zana
Eezha Ahsanat, Commentary on Hadith no. 6328)
And there are many reports of consensus from companions of the Prophet (salallahu alayhi wassalam) that the recitation of the verse was always meant to be abrogated but the ruling remained in effect. You can read them here.
This is the Sunni view.
I have linked the sources wherever I used stuff from them.
Thanks to goldPseudo for helping me edit the answer.