It's literal.
The answer the Angels give in 34:41 is a direct answer to the literal question asked in 34:40:
ويوم يحشرهم جميعا ثم يقول للملائكة أهؤلاء إياكم كانوا يعبدون ("And [mention] the Day when He will gather them all and then say to the angels, 'Did these [people] used to worship you?').
The 'people' being referred to are those who claimed to worship the Angels and that's why Allah (ﷻ) will bring these people & the Angels all together and He will ask the Angels the literal question - did they worship you truly? The question is meant to expose these people, from the mouths of the very beings (i.e. Angels) who they took as objects of worship.
To understand the answer of the Angels, you first have to understand what "worship" is. Worship includes obedience. You are worshiping the one whose orders you are following. Even if these people made Angels the 'object/image' of their worship, whose orders were they following? Did the Angels ask them to worship them (Angels)? No! So these people were following the orders of someone else.
So the Angels will answer that these people did not actually worship them (i.e., Angels) because the Angels never call to worship of anything other than Allah (ﷻ). Rather, they point out it was the devils (Shayateen) who they worshipped because it was the devils telling them to worship other than Allah (ﷻ). So it is a literal answer.
السلام عليكم ورحمة الله وبركاته