Yes that's true. But men shouldn't take it literally. There must be kindness and understanding. Islam ordered the husband to be nice and gentle and understanding. Mohammad (pbuh) said:
إن من أكمل المؤمنين إيمانا أحسنهم خلقا وألطفهم بأهله
“Indeed, the believers with the most complete faith are those with the most excellent character and are most kind to their women.” 
And in Quran:
يَا أَيُّهَا الَّذِينَ آمَنُواْ لاَ يَحِلُّ لَكُمْ أَن تَرِثُواْ النِّسَاء كَرْهًا وَلاَ تَعْضُلُوهُنَّ لِتَذْهَبُواْ بِبَعْضِ مَا آتَيْتُمُوهُنَّ إِلاَّ أَن يَأْتِينَ بِفَاحِشَةٍ مُّبَيِّنَةٍ وَعَاشِرُوهُنَّ بِالْمَعْرُوفِ فَإِن كَرِهْتُمُوهُنَّ فَعَسَى أَن تَكْرَهُواْ شَيْئًا وَيَجْعَلَ اللَّهُ فِيهِ خَيْرًا كَثِيرًا
O you who have believed, it is not lawful for you to inherit women by compulsion. And do not make difficulties for them in order to take [back] part of what you gave them unless they commit a clear immorality. And live with them in kindness. For if you dislike them - perhaps you dislike a thing and Allah makes therein much good.[4:19]
And guess what did Mohammed (pbuh) said in his Last speech to Muslims which means it is important:
ألا واستوصوا بالنساء خيراً فإنما هن عوان عندكم.
"Treat your women well. They are as captives in your possession." [Ref]
Also, Muslim men have more responsibilities towards women than women towards men. He is responsible of protecting them, providing (food, clothes, money..etc) and being kind to them. In addition to that, Muslims are strictly prohibited from oppressing any one, so what about his own wife?. Also, This is something which is a priority for most men but not most women (Usually), and it affects the stability of the relationship. Obviously, the man has to judge his situation and see if the timing is mutually conducive.