The music executive was teamed with Abdul on the judging panel of American Idol when it launched back in 2002, and currently works alongside her on X Factor.
Speaking to Barbara Walters about how their relationship had changed, he said: "We obviously like each other and people can see that. We didn't like each other at the beginning.
"The truth was on the first two seasons [of American Idol] we couldn't travel on the same plane together. We really disliked each other that much."
When asked where the dislike had stemmed from, Cowell replied: "I think it was sexual tension... on her part."
As for whether he ever considered acting on it, he laughed: "I considered it but then I thought, 'I don't think the after would be as good as the before'."
Cowell was speaking to Walters as part of Barbara Walters presents the 10 Most Fascinating People of the Year, which airs tonight (Wednesday, December 14) at 9.30pm ET.
Other stars named in the annual list include Kim Kardashian, Katy Perry, Pippa Middleton, Donald Trump and Eric Stonestreet.
Watch a preview of Simon Cowell talking with Barbara Walters below:
Abdul left American Idol in 2009, while Cowell departed the singing competition one year later in order to launch The X Factor in America, inviting Abdul to join him on the panel.
LA Reid and Nicole Scherzinger make up the four-panel judging team.
X Factor's final four contestants will take to the stage for their semi-final performances on tonight's live show, hoping to earn a place in next week's live finale where the $5 million recording contract prize will be awarded.
The contestants still in the running are: Melanie Amaro (mentored by Cowell), Josh Krajcik (mentored by Scherzinger), Chris Rene and Marcus Canty (mentored by Reid).
X Factor airs Wednesdays and Thursdays at 8/7c on Fox.
> Nicole Scherzinger: 'Audience booing was devastating'
> Read Digital Spy's full X Factor headlines