Spears, who joined the show's judging panel for this season, was "kind of fearless" during auditions, according to the mogul.
"You know how when you buy a dog, you expect it to lick you, and then it turns out to bite you?" Us Weekly quotes him as saying. "It was rather like that.
"It was a real surprise. She's really, really difficult to please. But in a way I think it made it more interesting because she just wouldn't say yes for people to like her. And when you do these auditions in front of 5,000 people, they boo you."
He added: "In the past, we've had judges who never say no. She can't say no quick enough on this show. She's quite difficult to please. But she's a fascinating person and I got to like her a lot."
Spears recently admitted that she suffered from panic attacks on her first day of judging The X Factor USA.
The show returns with its new season on Wednesday (September 12) on Fox.
Watch a promo for The X Factor USA featuring Spears below: