All perfect praise be to Allah
I will try to answer as per my knowledge, if anything in my answer is wrong, let me know i will update the answer. Being said that, let's move to the answer.
Let's first quote a hadees,
Anas ibn Malik reported: The Messenger of Allah, peace and blessings
be upon him, was in the company of one of his wives when a man passed
by them. The Prophet called to him and when he came, the Prophet said,
“She is my wife.” The man said, “O Messenger of Allah, I do not doubt
you in the least.” The Prophet said, “Verily, Satan flows through the
human being like blood.”
Sahih Muslim 2174
Now, this hadees has 2 important points in it, first is that shayateen flows in our body, and when something is in our body like blood, it will obviously know everything about our body. But keep in mind, not all shayateen runs in our body, as described in another hadees.
It was narrated that ‘Abd-Allaah ibn Mas’ood said: The Messenger of
Allaah (peace and blessings of Allaah be upon him) said: “There is no
one among you but a companion from among the jinn has been assigned to
him.” They said, “Even you, O Messenger of Allaah?’ He said, “Even me,
but Allaah helped me with him and he became Muslim (or: and I am safe
from him), so he only enjoins me to do that which is good.”
According to another report, “… There is assigned to him a companion
from among the jinn and a companion from among the angels.”
Narrated by Muslim, 2814
This companion(the devil one), which is commonly referred to as "Qareen"(arabic word meaning companion) is devil and it is directly responsible for all the wasawas and bad things in your mind, like Allah says in quran:
“His companion (qareen) will say: ‘Our Lord! I did not push him to
transgression, (in disbelief, oppression, and evil deeds), but he was
himself in error far astray.’
Allaah will say: ‘Dispute not in front of Me, I had already in advance
sent you the threat.
The Sentence that comes from Me cannot be changed, and I am not unjust
to the slaves’”
[Qaaf 50:27-29]
and
‘And Shaytaan (Satan) will say when the matter has been decided:
“Verily, Allaah promised you a promise of truth. And I too promised
you, but I betrayed you. I had no authority over you except that I
called you, and you responded to me. So blame me not, but blame
yourselves. I cannot help you, nor can you help me. I deny your former
act in associating me (Satan) as a partner with Allaah (by obeying me
in the life of the world). Verily, there is a painful torment for the
Zaalimoon (polytheists and wrongdoers).”’
[Ibraaheem 14:22]
And there are other verses like these in quran. The point i want to make is, that their is a devil associated with each person and it runs like blood in veins. Now, this does not mean we are alone in fighting it, as mentioned in above hadees, their is also an angel who is there to protect us. Now if we do things which make angel feel bad, it will move a little away from you, because those things kind of torture him, and shatan will be closer and happy. And if you do good things, which are a torture to shatan, shatan will feel pain and will be a little away, and angel will be happy and will be more close to you. This explanation is what i have heard from some big scholars and this totally makes sense after all above ahadees and ayahs.
Now, the second thing which the first hadees tells us is that prophet told the man that she is my wife, and he said i don't doubt you, but prophet saying "Satan flows through the human being like blood" means that it is satan who whispers and put bad thoughts in the mind of people. And prophet feared that satan might had whispered in his ear too, that's why he explained.
Now, we know that satan runs like blood in veins, and it can whisper in our mind, and it running like blood implies he would know anything in the mind or heart of the man too...
Now, this is not in contradiction with the verse you quoted. Because, firstly, jinns don't know the future, and every jinn don't know about the heart of every person, he knows about the person he is attached to only. So, hidden is still hidden and he knows what Allah has enabled him to know. There is no contradiction, we must understand that the definition of hidden is not same for everyone, for example if I don't know what's inside some room, it's possible some other person who has seen inside of the room knows what's inside, so we can't say he knows all the hidden things, but yes, he know something which is hidden for us.
All your sub-questions were on supposition that jins cannot read our thoughts, but as i explained, they do. So, i suppose their is nothing to clear in them.
One more thing i should mention is, that the people who are attached to spiritual healing and stuff like that like the persons who are possessed by jinns or devils, they know that the jinn inside the body knows everything a man is thinking or seeing or feeling, but the external jinns don't know it, and sometimes internal jinns communicate with external ones to let them know about the person. But external ones don't know anything like that themselves, they are just told sometimes.
Read sources of whispers/wasawas.