According to this page from the Book الأشباه والنظائر by Al-Suyuti:
القاعدة الثانية إذا اجتمع الحلال والحرام غلب الحرام وأورده جماعة حديثا بلفظ { ما اجتمع الحلال والحرام إلا غلب الحرام الحلال } . قال الحافظ أبو الفضل العراقي : ولا أصل له ، وقال السبكي في الأشباه والنظائر نقلا عن البيهقي : هو حديث رواه جابر الجعفي ، رجل ضعيف ، عن الشعبي عن ابن مسعود ، وهو منقطع . قلت : وأخرجه من هذا الطريق عبد الرزاق في مصنفه . وهو موقوف على ابن مسعود لا مرفوع .
Rule #2: If Halal and Haraam are mixed then Haram overwhelms. Hadith scholars mentioned the rule as a Hadith like this: {{ Nothing is mixed between that which is halal and that which is haraam, except in which the haraam overwhelms the halal }}. Hadith scholar Abu-Alfadhl Al-Iraqi said: this hadith has no origin (i.e, fabricated); While Al-Sabki in his book الأشباه والنظائر1 narrated that Al-Baihaqi said about it: A hadith narrated by Jabir Al-Ja’fi, a weak (untrustworthy) man, by Al-Sha'bi by Ibn Mas'ud, and it's narration chain is broken. And I (Al-Suyuti) say: Abdulrazzak also reported the hadith by the same people in his Hadith book; and it's a Mawkoof موقوف on Ibn Mas'ud and not Marfoo' (not a narration of the Prophet).
[1]: There are many books named الأشباه والنظائر, one by Al-Suyuti (and the quote above is from it), and another by Al-Sabki, and others.
Also, scholar زين الدين ابن نجيم says something very similar in this page of his Book الأشباه والنظائر على مذاهب أبي حنيفة النعمان, and adds:
وذكره الزيلعي شارح الكنز في كتاب الصيد مرفوعا.
And Imam Zaila'ii mentioned it as a Marfoo' hadith (a narration of the Prophet) in his book كتاب الصيد.
And according to this fatwa from islamweb.net, Imam Ibn Muflih said that the this hadith has no origin (i.e, fabricated).
So in summary: this hadith is either weak or fabricated, it's narrators chain is broken and it's not a narration of the Prophet, but rather a narration of Ibm Mas'ud.
However, it's worthy mention that the rule itself appears to be accepted and agreed upon by most scholars, especially those who I have quoted from their books.