Tom Bower's unauthorised biography The Intimate Life Of Simon Cowell claimed that the pair, who originally denied rumors in 2008, were 'genuinely in love'.
PR guru Clifford told Exclusive TV: "Yes - he did have a relationship with Dannii."
The 68-year-old was quick to defend Cowell by insisting that the star has never spoken publicly about his relationships, adding: "Everyone who knows Simon knows he keeps his private life as private as possible. But [it's] the truth."
Criticising Bower, Clifford added: "It was Simon's choice, but my attitude is you don't talk to someone like Tom Bower, he's only really looking for the controversial stuff. It was a mistake."
Minogue, who appeared on the X Factor panel alongside Cowell between 2007 and 2010, is reportedly distraught by the revelation. The 40-year-old split up with rugby player partner Kris Smith earlier this month.