From Tafsir Muhammad Asad:
إِلَّا عَلَىٰ أَزْوَاجِهِمْ أَوْ مَا مَلَكَتْ أَيْمَانُهُمْ فَإِنَّهُمْ غَيْرُ مَلُومِينَ
Quran 23:6
Lit., "or those whom their right hands possess" (aw ma malakat aymanuhum). Many of the commentators assume unquestioningly that this relates to female slaves, and that the particle aw ("or") denotes a permissible alternative. This interpretation is, in my opinion, inadmissible inasmuch as it is based on the assumption that sexual intercourse with ones female slave is permitted without marriage: an assumption, which is contradicted by the Qur’an itself (see 4:3, 24, 25 and 24:32, with the corresponding notes). Nor is this the only objection to the above-mentioned interpretation. Since the Qur’an applies the term ‘‘believers" to men and women alike, and since the term azwaj ("spouses"), too, denotes both the male and the female partners in marriage, there is no reason for attributing to the phrase ma malakat aymanuhum the meaning of "their female slaves’’; and since, on the other hand, it is out of the question that female and male slaves could have been referred to here it is obvious that this phrase does not relate to slaves at all, but has the same meaning as in 4:24 - namely, "those whom they rightfully possess through wedlock (see note 26 on 4:24) - with the significant difference that in the present context this expression relates to both husbands and wives, who "rightfully possess" one another by virtue of marriage. On the basis of this interpretation, the particle aw which precedes this clause does not denote an alternative ("or") but is, rather, in the nature of an explanatory amplification, more or less analogous to the phrase "in other words" or "that is", thus giving to the whole sentence the meaning, "save with their spouses - that is, those whom they rightfully possess [through wedlock]", etc. (Cf. a similar construction 25:62 - ‘‘for him who has the will to take thought -that is [lit., "or"], has the will to be grateful".)
Asad is saying that the verse means 'spouses'.
However, it can be argued that it means 'wives'.
The verses before this relate to believers in general:
قَدْ أَفْلَحَ الْمُؤْمِنُونَ
الَّذِينَ هُمْ فِي صَلَاتِهِمْ خَاشِعُونَ
وَالَّذِينَ هُمْ عَنِ اللَّغْوِ مُعْرِضُونَ
وَالَّذِينَ هُمْ لِلزَّكَاةِ فَاعِلُونَ
وَالَّذِينَ هُمْ لِفُرُوجِهِمْ حَافِظُونَ
The reason people refer to the verse as 'wives' is because of 'right hand possess' which is meaning the female slave captives.
The only way to get a slave is from booty of war and women don't necessarily fight and be soldiers.
Ibn ‘Umar said:
“I presented myself to the Messenger of Allaah (peace and blessings of Allaah be upon him) on the day of Uhud when I was fourteen years old, and he did not let me join the fighting.” Agreed upon. … With regard to being male, this is stipulated because of the report narrated from ‘Aa’ishah who said: “O Messenger of Allaah, do women have to engage in jihad? He said: “Jihad in which there is no fighting: Hajj and ‘Umrah.” And because women are not able to fight because they are (physically) weak.
(al-Mughni, 9/163)
Majority of people say that it means 'wives' in this case.
With regards to 'right hand possess', N.J. Dawood translates its as:
'those whom you own as slaves'
Therefore a women can own a slave. However, this wouldn't refer to women.
It is haraam for a slave to marry his mistress. Ibn al-Mundhir said: The scholars are unanimously agreed that marriage of a woman to her slave is invalid. Al-Athram narrated, with his isnaad from Abu’z-Zubayr, who said: I asked Jaabir about a slave marrying his mistress, and he said: A woman came to ‘Umar ibn al-Khattaab, when we were in al-Jaabiyyah. She had married her slave, and ‘Umar rebuked her and thought of stoning her, and he said: He is not permissible for you.
If a woman owns her husband [i.e., if he is a slave or she bought him, for example], her marriage becomes invalid.
Al-Mughni (7/113)
So if a women did own a male slave, she can't have sexual relations with him or marry him.
The verse says:
وَالَّذِينَ هُمْ لِفُرُوجِهِمْ حَافِظُونَ
And those who guard their private parts
Meaning those who refrain from sex and the verse which you quote, except from their wives and female slaves.
If a women cannot have sexual relations with her male slave, than this verse doesn't apply to women and therefore would be about men and أَزْوَاجِهِمْ would translate into 'wives'.
In conclusion, it would be best to translate this into 'wives' based on the reasons above.
And Allah knows best
