First of all, if you are asking this question because you want to verify the authenticity of a man who claims to be a prophet, then don't. There are no prophets anymore, as Prophet Muhammad was the last of the Prophets.
This may not be a complete answer, but I'll put what I can think of.
Looking into Quran, we can find verses where Prophets are trying to convince people that they're prophets, and sometimes people weren't convinced directly but rather needed or asked for some proofs. On the type of proofs, I can see:
- Miracles, i.e extra ordinary things that people cannot regenerate on their own at their will.
- Reason, convincing and attitude.
And sometimes a combination of both, sometimes with a challenge.
Let's read the following verses:
وَقَالَ مُوسَىٰ يَا فِرْعَوْنُ إِنِّي رَسُولٌ مِّن رَّبِّ الْعَالَمِينَ * حَقِيقٌ عَلَىٰ أَن لَّا أَقُولَ عَلَى اللَّهِ إِلَّا الْحَقَّ ۚ قَدْ جِئْتُكُم بِبَيِّنَةٍ مِّن رَّبِّكُمْ فَأَرْسِلْ مَعِيَ بَنِي إِسْرَائِيلَ * قَالَ إِن كُنتَ جِئْتَ بِآيَةٍ فَأْتِ بِهَا إِن كُنتَ مِنَ الصَّادِقِينَ * فَأَلْقَىٰ عَصَاهُ فَإِذَا هِيَ ثُعْبَانٌ مُّبِينٌ * وَنَزَعَ يَدَهُ فَإِذَا هِيَ بَيْضَاءُ لِلنَّاظِرِينَ * قَالَ الْمَلَأُ مِن قَوْمِ فِرْعَوْنَ إِنَّ هَٰذَا لَسَاحِرٌ عَلِيمٌ * يُرِيدُ أَن يُخْرِجَكُم مِّنْ أَرْضِكُمْ ۖ فَمَاذَا تَأْمُرُونَ * قَالُوا أَرْجِهْ وَأَخَاهُ وَأَرْسِلْ فِي الْمَدَائِنِ حَاشِرِينَ * يَأْتُوكَ بِكُلِّ سَاحِرٍ عَلِيمٍ * وَجَاءَ السَّحَرَةُ فِرْعَوْنَ قَالُوا إِنَّ لَنَا لَأَجْرًا إِن كُنَّا نَحْنُ الْغَالِبِينَ * قَالَ نَعَمْ وَإِنَّكُمْ لَمِنَ الْمُقَرَّبِينَ * قَالُوا يَا مُوسَىٰ إِمَّا أَن تُلْقِيَ وَإِمَّا أَن نَّكُونَ نَحْنُ الْمُلْقِينَ * قَالَ أَلْقُوا ۖ فَلَمَّا أَلْقَوْا سَحَرُوا أَعْيُنَ النَّاسِ وَاسْتَرْهَبُوهُمْ وَجَاءُوا بِسِحْرٍ عَظِيمٍ * وَأَوْحَيْنَا إِلَىٰ مُوسَىٰ أَنْ أَلْقِ عَصَاكَ ۖ فَإِذَا هِيَ تَلْقَفُ مَا يَأْفِكُونَ * فَوَقَعَ الْحَقُّ وَبَطَلَ مَا كَانُوا يَعْمَلُونَ * فَغُلِبُوا هُنَالِكَ وَانقَلَبُوا صَاغِرِينَ * وَأُلْقِيَ السَّحَرَةُ سَاجِدِينَ * قَالُوا آمَنَّا بِرَبِّ الْعَالَمِينَ * رَبِّ مُوسَىٰ وَهَارُونَ * قَالَ فِرْعَوْنُ آمَنتُم بِهِ قَبْلَ أَنْ آذَنَ لَكُمْ ۖ إِنَّ هَٰذَا لَمَكْرٌ مَّكَرْتُمُوهُ فِي الْمَدِينَةِ لِتُخْرِجُوا مِنْهَا أَهْلَهَا ۖ فَسَوْفَ تَعْلَمُونَ * لَأُقَطِّعَنَّ أَيْدِيَكُمْ وَأَرْجُلَكُم مِّنْ خِلَافٍ ثُمَّ لَأُصَلِّبَنَّكُمْ أَجْمَعِينَ * قَالُوا إِنَّا إِلَىٰ رَبِّنَا مُنقَلِبُونَ
Sahih International Translation
And Moses said, "O Pharaoh, I am a messenger from the Lord of the worlds * [Who is] obligated not to say about Allah except the truth. I have come to you with clear evidence from your Lord, so send with me the Children of Israel." * [Pharaoh] said, "If you have come with a sign, then bring it forth, if you should be of the truthful." * So Moses threw his staff, and suddenly it was a serpent, manifest. * And he drew out his hand; thereupon it was white [with radiance] for the observers. * Said the eminent among the people of Pharaoh, "Indeed, this is a learned magician * Who wants to expel you from your land [through magic], so what do you instruct?" * They said, "Postpone [the matter of] him and his brother and send among the cities gatherers * Who will bring you every learned magician." * And the magicians came to Pharaoh. They said, "Indeed for us is a reward if we are the predominant." * He said, "Yes, and, [moreover], you will be among those made near [to me]." * They said, "O Moses, either you throw [your staff], or we will be the ones to throw [first]." * He said, "Throw," and when they threw, they bewitched the eyes of the people and struck terror into them, and they presented a great [feat of] magic. * And We inspired to Moses, "Throw your staff," and at once it devoured what they were falsifying. * So the truth was established, and abolished was what they were doing. * And Pharaoh and his people were overcome right there and became debased. * And the magicians fell down in prostration [to Allah ]. * They said, "We have believed in the Lord of the worlds, * The Lord of Moses and Aaron." * Said Pharaoh, "You believed in him before I gave you permission. Indeed, this is a conspiracy which you conspired in the city to expel therefrom its people. But you are going to know. * I will surely cut off your hands and your feet on opposite sides; then I will surely crucify you all." * They said, "Indeed, to our Lord we will return.
In these verses, Pharaon and his people first required Moses to show the proofs of prophecy that he claimed that he had, and so he did. He showed them how his staff became a serpent and how his hand turned white with radiance. That wasn't enough for them, though, and they invited him for a challenge, which he accepted. They gathered the most powerful magicians in the region to challenge him. The magicians cast their magic, and Moses threw his staff which ultimately devoured all of their magic. The magicians were very skilled at magic and could realize it very well, and they knew that what they saw, what Moses did, was really not magic, but a true miracle of a prophet, and hence they readily believed that he is a true Prophet, without fearing the horrible punishment of Pharaon.
2) Reason and Convincing
Let's read the following verses:
وَلَقَدْ آتَيْنَا إِبْرَاهِيمَ رُشْدَهُ مِن قَبْلُ وَكُنَّا بِهِ عَالِمِينَ * إِذْ قَالَ لِأَبِيهِ وَقَوْمِهِ مَا هَٰذِهِ التَّمَاثِيلُ الَّتِي أَنتُمْ لَهَا عَاكِفُونَ * قَالُوا وَجَدْنَا آبَاءَنَا لَهَا عَابِدِينَ * قَالَ لَقَدْ كُنتُمْ أَنتُمْ وَآبَاؤُكُمْ فِي ضَلَالٍ مُّبِينٍ * قَالُوا أَجِئْتَنَا بِالْحَقِّ أَمْ أَنتَ مِنَ اللَّاعِبِينَ * قَالَ بَل رَّبُّكُمْ رَبُّ السَّمَاوَاتِ وَالْأَرْضِ الَّذِي فَطَرَهُنَّ وَأَنَا عَلَىٰ ذَٰلِكُم مِّنَ الشَّاهِدِينَ * وَتَاللَّهِ لَأَكِيدَنَّ أَصْنَامَكُم بَعْدَ أَن تُوَلُّوا مُدْبِرِينَ * فَجَعَلَهُمْ جُذَاذًا إِلَّا كَبِيرًا لَّهُمْ لَعَلَّهُمْ إِلَيْهِ يَرْجِعُونَ * قَالُوا مَن فَعَلَ هَٰذَا بِآلِهَتِنَا إِنَّهُ لَمِنَ الظَّالِمِينَ * قَالُوا سَمِعْنَا فَتًى يَذْكُرُهُمْ يُقَالُ لَهُ إِبْرَاهِيمُ * قَالُوا فَأْتُوا بِهِ عَلَىٰ أَعْيُنِ النَّاسِ لَعَلَّهُمْ يَشْهَدُونَ * قَالُوا أَأَنتَ فَعَلْتَ هَٰذَا بِآلِهَتِنَا يَا إِبْرَاهِيمُ * قَالَ بَلْ فَعَلَهُ كَبِيرُهُمْ هَٰذَا فَاسْأَلُوهُمْ إِن كَانُوا يَنطِقُونَ * فَرَجَعُوا إِلَىٰ أَنفُسِهِمْ فَقَالُوا إِنَّكُمْ أَنتُمُ الظَّالِمُونَ * ثُمَّ نُكِسُوا عَلَىٰ رُءُوسِهِمْ لَقَدْ عَلِمْتَ مَا هَٰؤُلَاءِ يَنطِقُونَ * قَالَ أَفَتَعْبُدُونَ مِن دُونِ اللَّهِ مَا لَا يَنفَعُكُمْ شَيْئًا وَلَا يَضُرُّكُمْ * أُفٍّ لَّكُمْ وَلِمَا تَعْبُدُونَ مِن دُونِ اللَّهِ ۖ أَفَلَا تَعْقِلُونَ * قَالُوا حَرِّقُوهُ وَانصُرُوا آلِهَتَكُمْ إِن كُنتُمْ فَاعِلِينَ * قُلْنَا يَا نَارُ كُونِي بَرْدًا وَسَلَامًا عَلَىٰ إِبْرَاهِيمَ * وَأَرَادُوا بِهِ كَيْدًا فَجَعَلْنَاهُمُ الْأَخْسَرِينَ * وَنَجَّيْنَاهُ وَلُوطًا إِلَى الْأَرْضِ الَّتِي بَارَكْنَا فِيهَا لِلْعَالَمِينَ
Sahih International Translation
And We had certainly given Abraham his sound judgement before, and We were of him well-Knowing * When he said to his father and his people, "What are these statues to which you are devoted?" * They said, "We found our fathers worshippers of them." * He said, "You were certainly, you and your fathers, in manifest error." * They said, "Have you come to us with truth, or are you of those who jest?" * He said, "[No], rather, your Lord is the Lord of the heavens and the earth who created them, and I, to that, am of those who testify. * And [I swear] by Allah , I will surely plan against your idols after you have turned and gone away." * So he made them into fragments, except a large one among them, that they might return to it [and question]. * They said, "Who has done this to our gods? Indeed, he is of the wrongdoers." * They said, "We heard a young man mention them who is called Abraham." * They said, "Then bring him before the eyes of the people that they may testify." * They said, "Have you done this to our gods, O Abraham?" * He said, "Rather, this - the largest of them - did it, so ask them, if they should [be able to] speak." * So they returned to [blaming] themselves and said [to each other], "Indeed, you are the wrongdoers." * Then they reversed themselves, [saying], "You have already known that these do not speak!" * He said, "Then do you worship instead of Allah that which does not benefit you at all or harm you? * Uff to you and to what you worship instead of Allah . Then will you not use reason?" * They said, "Burn him and support your gods - if you are to act." * Allah said, "O fire, be coolness and safety upon Abraham." * And they intended for him harm, but We made them the greatest losers. * And We delivered him and Lot to the land which We had blessed for the worlds.
In these verses, Prophet Ibrahim tried to convince his people that statuses (idols) are not to be worshiped and that Allah is the true and only god. They weren't convinced though, so he decided to prove that with reason. He waited until they left the city (for a holiday of one or more days) and destroyed all the idols except the largest one. When they returned they were shocked to see their idols destroyed and asked him: "are you the one who destroyed our idols?", he denied that and said that the largest one (which he didn't destroyed) is the one who destroyed the others, and invited them to ask the idols and see if they can speak out. The people replied: "but you know the idols cannot speak out", and Ibrahim replied that you can see now how you worship idols that have no benefit or harm, and cannot even defend themselves. The people were angered to find themselves wrong and to see their idols destroyed, and to avenge that they threw Ibrahim to a great fire to burn him, and then a miracle occurred as Ibrahim came out of the fire without being harmed at all. This is an example of the Reason+Miracle combination.
Another example of reason and attitude as a proof of prophecy can probably be found in [Surat Yā-Sīn:13-27].
Also, a very big example of a miracle and challenge is the Quran, which is an eternal miracle because none of the beings can author something like it. Being miraculous, Prophet Muhammad challenged the Arabs, who were very skilled at poetry and eloquence, to be able to author a few sentences that come close to Quran in matters of eloquence and correctness, but they weren't able to do so.