If the parents teach their children to prioritize worldly success over success in the hereafter or if they do not pay much attention about their children, whether they are committing sins or not praying, especially if they do not encourage their children to pray or teach them the importance of prayer and do not pay attention to it, In such a case, what is the rule of Islam regarding these parents?
Our Lord, and make us Muslims [in submission] to You and from our descendants a Muslim nation [in submission] to You. And show us our rites [of worship] and accept our repentance. Indeed, You are the Accepting of Repentance, the Merciful [2:128]
Islam is a religion that has been perfected, every aspect of our mundane lives has guidance. nothing is left out and we are never left without knowing.
from the time immemorial, mankind has been created in such a way that in reality we look out for each other, we pass down our traits, we influence the one's to come after us and we shape what has been entrusted to us. the evidence for this is in the verse; And (remember) as your Lord said to the Angels, "Surely I am making in the earth a successor"(2:30).
as we succeeed and become successors, the responsibility or the trust known as parent has become what makes the real differences in our lives and our ultimate end.
FROM THE QURAN AND SUNNAH we see so much emphasis laid on the role of parents and explicit directions on how to raise children. the obligation of parenthood is established in the following verse;
O you who have believed, protect yourselves and your families from a Fire whose fuel is people and stones, over which are [appointed] angels, harsh and severe; they do not disobey Allah in what He commands them but do what they are commanded.(66:6)
regarding this verse, it was reported;
Al Fakhr ar-Raazi said in his tafsir that it means protect yourselves by abstaining from what Allah has forbidden. Others said it means to protect yourselves by abandoning sin and working righteousness and to hold your families accountable as you hold yourselves accountable.
it is incumbent upon us, specially as parents to save our families from all kinds of destruction. beloved prophetﷺ has given very clear instructions regarding all the aspects of parenthood. in regard to the context of your question, ill quote the following hadith as the baseline to move forward;
Narrated Abu Huraira: The Prophet (ﷺ) said, "Every child is born with a true faith of Islam (i.e. to worship none but Allah Alone) and his parents convert him to Judaism or Christianity or Magianism, as an animal delivers a perfect baby animal. Do you find it mutilated?" (Bukhari)
for a short answer, this hadith itself is enough. it is an obligation upon every parent to raise their children in the best manner and to guide them in the best direction. as parents, we are responsible for the well-being of the children. this is stressed and clarified in another narration;
Narrated Abdullah bin Umar: Allah's Messenger (ﷺ) said, "Surely! Everyone of you is a guardian and is responsible for his charges: The Imam (ruler) of the people is a guardian and is responsible for his subjects; a man is the guardian of his family (household) and is responsible for his subjects; a woman is the guardian of her husband's home and of his children and is responsible for them; and the slave of a man is a guardian of his master's property and is responsible for it. Surely, everyone of you is a guardian and responsible for his charges" (Bukhari)
the rights of parents upon thier children has to fullfilled to the best of their ability. it is a must upon the parents to teach the child what the priorities must be. if the child is going astray, even in the smallest of things, they must be corrected. and regarding prayers and other obligations in deen, the instructions of our beloved prophetﷺ is very clear and stern, the evidence for this is the following hadith;
Narrated Abdullah ibn Amr ibn al-As: The Messenger of Allah (ﷺ) said: Command your children to pray when they become seven years old, and beat them for it (prayer) when they become ten years old; and arrange their beds (to sleep) separately. (abu dawud)
FURTHER ELOBORATING on part of the question about whether they are committing sins or not praying, especially if they do not encourage their children to pray or teach them the importance of prayer and do not pay attention to it;
it is narrated in a hadith the importance of enjoining prayer upon the children from the very young age.
Narrated Abdullah ibn Amr ibn al-As: The Messenger of Allah (ﷺ) said: Command your children to pray when they become seven years old, and beat them for it (prayer) when they become ten years old; and arrange their beds (to sleep) separately [abu dawud]
to emphasis more on this responsibility, ill quote what some scholars have said regarding this;
Al-Nawawi said: The father must discipline his child and teach him what he needs to know of religious duties. This teaching is obligatory upon the father and all those in charge of children before the child reaches the age of adolescence.
Al-Shaafa'i and his companions said: This teaching is also obligatory upon the mother, if there is no father, because it is part of the child’s upbringing and they have a share of that and the wages for this teaching may be taken from the child’s own wealth. If the child has no wealth then the one who is obliged to spend on him may spend on his education, because it is one of the things that he needs. And Allah knows best.
Ibn Qayyim said Whoever neglects to teach his child that which will benefit him, and leaves him without any guidance, has wronged him gravely. Most children’s deviance is because of their parents and the parents’ neglect and failure to teach them the obligatory religious duties and the sunnah of the Prophetﷺ, so they neglected them when they were small.
that the father will be asked about his child before the child is asked about his father.
based on all the available evidences, intentional failure and negligence of the parents on the part of good parenting is considered as not properly following the Quran and Sunnah, thus becoming all the more closer to a grave sin.
TO CONCLUDE i would like to quote an incident during the caliphate of Umar (RA), as this beautifully sums up the rights of parents to their children as well as vice versa, in sha Allah. It is related that a man once came to Umar Ibn Al-Khattab, the second Khalifah of Islam, may Allah be pleased with him, complaining of his son's disobedience to him. Umar summoned the boy and spoke of his disobedience to his father and his neglect of his rights. The boy replied:
“O Amir al-Mu'minin (Prince of believers)! Hasn't a child rights over his father?” “Certainly”, replied Umar. “What are they, Amir al-Mu'minin?” “That he should choose his mother, give him a good name and teach him the Book (the Qur'an).” “O Amir al-Mu’minin! My father did nothing of this. My mother was a Magian (fire worshipper). He gave me the name of Julalan (meaning dung beetle or scarab) and he did not teach me a single letter of the Qur'an.” Turning to the father, Umar, may Allah be pleased with him, said: “You have come to me to complain about the disobedience of your son. You have failed in your duty to him before he has failed in his duty to you; you have done wrong to him before he has wronged you.”
with the above mentioned few and numerous other reports/narrations on the obligations and responsibility of parents, its clear that it's a sin to neglect the correct upbringing of your children. when Allah azzawajjal entrusts something to us, we have to do our best to try to do that trust as it should be done.
i would like to mention a verse which i feel is very relevant to this question;
And those who believe and whose families follow them in Faith, to them shall We join their families: Nor shall We deprive them (of the fruit) of aught of their works: (Yet) is each individual in pledge for his deeds. [52:21]
may Allah the mighty and sublime grant us mercy and guide us to the straight path.
Allah knows best