The hadith you are looking for has been quoted in several sources, but the most complete chain appears in al-Bayhaqi's az-Zuhd al-Kabir الزهد الكبير, as a narration on the authority of Jabir, where al-Bayhaqi after quoting it says this sanad (chain) is weak:
أَخْبَرَنَا عَلِيُّ بْنُ أَحْمَدَ بْنِ عَبْدَانَ ، أَنْبَأَنَا أَحْمَدُ بْنُ عُبَيْدٍ ، حَدَّثَنَا تَمْتَامٌ ، حَدَّثَنَا عِيسَى بْنُ إِبْرَاهِيمَ ، حَدَّثَنَا يَحْيَى بْنُ يَعْلَى ، عَنْ لَيْثٍ ، عَنْ عَطَاءٍ ، عَنْ جَابِرٍ رَضِيَ اللَّهُ عَنْهُ , قَالَ : " قَدِمَ عَلَى رَسُولِ اللَّهِ صَلَّى اللَّهُ عَلَيْهِ وَسَلَّمَ قَوْمٌ غُزَاةٌ ، فَقَالَ صَلَّى اللَّهُ عَلَيْهِ وَسَلَّمَ : قَدِمْتُمْ خَيْرَ مَقْدَمٍ مِنَ الْجِهَادِ الأَصْغَرِ إِلَى الْجِهَادِ الأَكْبَرِ , قَالُوا : وَمَا الْجِهَادُ الأَكْبَرُ ؟ قَالَ : مُجَاهَدَةُ الْعَبْدِ هَوَاهُ " . هَذَا إِسْنَادٌ ضَعِيفٌ . (Source)
Al-Khatib al-Baghdadi has also quoted it in his Tarikh Baghdad (with an even weaker chain). The major issue in the chain above is the narrator Laith ibn abi Saleem al-Qurashi الليث بن أبي سليم القرشي who heard it via 'Ata' from Jabir and is considered as da'if (primarily because of tadlees, and making mistakes by merging ahadith). Ibn Rajab al-Hanbali -see here in Arabic- uttered the assumption that the qualification of al-Bayhaqi is referring to Tamtaam تَمْتَام who is Muhammad ibn Ghalib at-Tammaar محمد بن غالب التمار who with the age had issues (mistakes according to a-Daraqotni) in his narrations, while he basically is considered as trustworthy.
This hadith is considered as da'if by many scholars: al-Hafidh al-'Iraqi has considered it as such in his verification of the ahadith quoted in imam al-Ghazali's Ihya', as-Suyuti said he don't know about it being something attributed to the prophet(), Ibn Taymiyyah said it has no origin, al-Albani considered it as Munkar. ibn Hajar al-'Asqalani, ibn Rajab and an-Nasa-i held the opinion it is of the words of Ibrahim ibn abi 'Ablah إبراهيم بن أبي عبلة.
These qualifications can be found in Arabic here, on the website of sheikh al-Qaradawi, in this thread and in islamqa #127009.