These are a lot of questions in one post!
Let me try to answer them in a different order and with an introduction.
When was the sunnah or ahadith first compiled?
First of all, hadith was already gathered, written or memorized at the time of the prophet (). Some of the sahabah who have been said to have left written hadith in what we may call books are 'Abddullah ibn 'Amr ibn al'Aas (his collection was called as-Sahifa as-Sadiqah) and Anas ibn Malik...
Later this tradition was at least re-initiated by some tabi'iyn and one of the earliest hadith collections, that you may find on sunnah.com is imam Malik's al-Muwatta' which was originally compiled around 100 years after the death of the prophet().
See also What was the earliest Hadith Book / Collection to be recorded?
Some short introduction on concepts and apparent differences in books
As for the hadith collections the authors held different opinions and concepts:
For example, imam Malik would gather in his al-Muwatta' only reports related to fiqh topics, but he'd also consider sayings and verdicts of the sahabah and tabai'iyn in his book, so it is not only a book collecting sayings and doings of the prophet (). Imam Malik further, has chosen the most authentic reports -based on his own opinion-. That's why when it comes to the up to 700+ ahadith in l-Muwatta' you may find at least 500 out of it in each later hadith collection but reported via different students of imam Malik (so a slight difference in wording is possible since imam Malik kept working and updating his book till his death, this also why the different Muwatta' versions have a slight difference of amount in ahadith and reports). Since at his time hadith fabrication was at least not that much practiced in Hijaz imam Malik also accepted some kinds of ahadith we'd call da'if since they have disconnected narrator chain like marfo' and mursal ahadith.
Other authors collected all reports that at least were accepted in jurisprudence matters by some scholars like imam abu Dawood did in his sunan, a similar concept is present in other sunan collections including Jami' at-Tirmidhi, some of these authors also at least tried to exclude fabricated reports in their books. Especially the sunan books didn't report scholarly verdicts (be it from sahabah, tabi'yn or later scholars) solely, they rather reported them in form of a narration with a narrator chain.
A few authors have chosen to only quote sahih ahadith, meaning authentic ahadith that follows a predefined interpretation, on what is sahih according to the author the most prominent and most authentic books in this regard are the sahih's of the imams al-Bukhari and Muslim. Note that especially imam al-Bukhari emphasized in the book title that the reports that are musnad (meaning they have a connected narrator chain) are those he'd consider as sahih. Al-Bukhari for example wrote in his collection some kinds of introduction: verdicts or sayings of sahabah or verses of the qur'an that maybe related to the topic.
Most authors often left comments or statements on some reports that might be regarded as additional information.
Finally some authors only gathered all kinds of narrations like an encyclopedia the most prominent are the 3 books of imam at-Tabarani called al-M'ojam al-Awsat, al-Mo'jam al-Kabir and al-Mo'jam as-Saghir.
See also:
Is Musnad Ahmed a book of Sahih Ahaadith?
Why isn't Muwatta of Malik as famous as the six books of hadith??
Collection of ahadith arranged according to sahabi from whom the hadith is narrated
Is al Hakims Hadith collection Al Mustadrak reliable?
Sahih hadith
In the 3rd hijri century almost all reports of the sunnah of the prophet () had been gathered and compiled by different scholars. So that imam Ahmad and other scholars -who predated al-Bukhari and Muslim- of his time concluded that there are around 4400 sahih (authentic) hadith narrations since neither the sahih of imam al-Bukhari nor that of Muslim has 4400 of ahadith with different content and together they may reach an amount of around 3K (~2980 more exactly) sahih ahadith you may notice that other hadith collections may have the resting ~1400 ahadith which shall be authentic according to this view.
Before going ahead one must know that imam Ahmad was called an amir of Mu'minin in hadith he was a hafidh that had collected 600.000 different hadith reports (different here doesn't mean different in content only, but also in narrators of the narrator chain) out of which he has chosen less than 30.000 for his Musnad. Imam al-Bukhari gathered a similar high amount. While Imam Malik was said to have gathered over 10K reports just for the compilation of his al-Muwatta' out of which he has chosen ~2K over all imam Malik had gathered more than 100K of the hadith of the people of Hijaz and 10K only from ibn Shihaab az-Zuhri (one of his most prominent teachers) and he picked only the part of it he regarded the most authentic to compile it in his al-Muwatta' or which was reported via him in other hadith collections (since al-Muwatta' is mainly a compilation of reports on fiqh matters, while imam Malik also reported other kinds of ahadith).
So what I mean to say, is that almost any reliable hadith "content" should be found in imam Ahmad's al-Musnad, that's why scholars like him who traveled to gather hadith from the known narrators themselves have a good estimation of what is authentic and what isn't, and he showed this in some of his other books.
Further, to make clear what "all most all reports of the sunnah of the prophet" means if we take for example the hadith compilations of imam al-Bayhaqi (His Sunan and 4 more books, compiled in the 5th century) we may find that they gathered ~47K reports of which only 6% can be called ahadith out of which 55% is covered in the two sahihs, in other words, the main difference between later and earlier books are the chains of transmissions. For example, if we assume that imam Malik had only 5 students (in fact he had at least 100 who only narrated al-Muwatta' from him) who reported from him you may imagine that at the time of al-Bukhari and Muslim who transmitted his hadith via intermediate the paths of the narrator chains already doubled (since it is well known that both narrated via different intermediates)... so yes, with the time we have more ahadith in amount, but not necessarily in content.
Different hadith compilations have common and different content.
On the other hand, the above shows the need of different books because the content may be different.
For example, while al-Muwatta' has certainly a smaller content than that of both sahihs it includes 517 reports that were mentioned at least in one of them.
Furthermore, if we compared only both sahih books, we come to the following values:
- both books have ~7.5 of ahadith that can be called musnad (including repetitions). For example, all reports and introductions in Sahih al-Bukhari with all the chains of transmission come up to 9082 reports according to Arabic Wikipedia).
- Without repetition Sahih al-Bukhari 2602 (according to ibn Hajar) as for Sahih Muslim the value is 3033 according to Muhammad Fu'ad 'Abdal-baqi.
- The common part of ahadith which are in both Sahih's is ~2200 hadith. And seemingly 780 that are in either of both (note that this maybe inconsistent with the count of Muhammad Fu'ad 'Abdal-baqi)
Now let's have a small look at the additions made by other books
- Abu Dawood in his Sunan added ~2450 hadith (without repetitions) to the ahadith which are in either of both sahihs.
- At-Tirmidhi added ~1350 ahadith over the number of ahadith of the two sahihs and the Sunan of abu Dawood (all without repetitions).
...
From the above, we can see that each book even if after the two Sahih's and al-Muwatta' we have mainly an addition of rather weak reports, gives an addition to the most authentic reports and we must be aware that it could contain at least some of the 1.4K Sahih reports which are in neither of both sahihs.
See also:
Which ahadith quoted in al-Mustadrak are present in one or both sahihs?
Most hadith scholars had deep knowledge in hadith, why they quoted da'if narrations in their compilations then?
Some more sites I've consulted while answering (all in Arabic):
- Site of the contemporary scholar ibn Amin.
- Al-Jazeera.net, a Blog article on the amount of hadith discussing some claims of 'Adnan Ibraheem which seem to pretend that (or wonder why) the amount of hadith increased with times and expressed astonishment to the fact that there are only 4K+ sahih narratives, while the sunnah books cover more than 110K of ahadith.
