I haven't come across any reference mentioning a legal limitation on the number of slaves. The following are some references that negate any limit, the English text is my personal translation and may contain paraphrasing, mistakes and omissions.
ولا خلاف أن له أن يجمع من العدد من شاء بملك اليمين
There is no difference of opinion that one may keep whatever number they wish of slaves
— Tafsir al-Jassas
يعني السراري لأنه لا يلزم فيهن من الحقوق ما يلزم في الحرائر ، ولا قسم لهن ، ولا وقف في عددهن
Meaning female slaves, for regarding them the same rights are not obligatory as are the obligatory rights regarding free women. There is no equitable distribution between them and no restriction on their number
— Tafsir al-Baghawy
لا يتحدد ما يحل للرجل من السراري بأربع ولا بعدد معين
ولو كان عنده من الزوجات واحدة فأكثر إلى أربع أو لم يكن جاز له أن يتسرى بما شاء من الجواري
There is no limit on the number of female slaves. A man can keep as many slaves as he wants, be they four or more. Whether he has one to four wives or is not married at all. In all cases he can keep as many female slaves as he wants.
— الموسوعة الفقهية الكويتية
4:3 has not mentioned limits regarding the number slaves, while mentioning limits regarding the number of wives. So it is inferable that there is no limit as otherwise it would have been mentioned.
The verse has commanded to limit oneself to slaves when one can not do justice to even one wife. This means that a person does not have the same obligations to their slaves as they do to their wives.
The verse has limited the number of wives based on the husband's capacity to observe justice between them: He has to provide each with adequate maintenance and lodging, and is required to provide them with companionship and to equally divide his time between them. Having multiple wives makes it difficult to give each their share.
On the other hand, a person does not owe such obligations to a slave, so there is no limit on how many he can have.