The singers were officially confirmed to join LA Reid and Simon Cowell on the X Factor panel at the Fox Upfronts in New York yesterday (May 14).
Commenting on the new lineup of his rival show, Lythgoe said that although Spears and Lovato would have an "interesting" rapport with contestants, he didn't consider them to have enough "experience" in the music industry.
"I hadn't ever considered them," he told E! "I always like mentors who have a lot more experience.
"Having said that, these two ladies have had so much personal experience; it'll be interesting. I like someone who's been there and knows the whole gamut of what these kids are going through and help them through."
Commenting on the hiring of Spears and Lovato yesterday, Cowell said: "Britney remains one of the biggest stars in the world, she's talented, fascinating - and I believe she knows exactly how to spot The X Factor.
"Demi's had an amazing career in music, TV and film for someone her age. She's young, confident, and enthusiastic. I think it's really important that she speaks to our younger audience."
Spears is rumoured to have landed a $15m (£9.3m) fee for her appearance on The X Factor, while Lovato is reportedly being paid $1m (£622,000).
The pair are replacements for X Factor season one panelists Nicole Scherzinger and Paula Abdul.