The former panellist, who left The X Factor in 2010, has recently been linked with a return to the ITV1 reality show following the departure of Kelly Rowland.
However, Minogue - who is now a judge on Australia's Got Talent with former Westlife musician Brian McFadden and radio DJ Kyle Sandilands - has suggested that she does not plan a return to the UK in a magazine interview.
Minogue told the Australian edition of Marie Claire: "[My son] Ethan has his little routine now. He has swimming lessons. He has things that are important to him. I don't want to unsettle him to follow me around the globe. The guilt of going back to work floods you.
"I try to take it all in my stride, but when it affects my loved ones I feel an internal struggle with the love for my work. Fame and work have always had an impact on my family since I went into this business at seven. It's all I know."
Minogue recently separated from former partner Kris Smith, with whom she welcomed Ethan in July 2010.
Louis Walsh and Gary Barlow have both confirmed that they will return to The X Factor panel, with former N-Dubz member Tulisa Contostavlos expected to join them once again.
The fourth judging slot needs to be filled before May 23, when taped auditions in front of the programme's panel are due to begin in Liverpool.
Digital Spy readers overwhelmingly voted for Minogue as their preferred replacement for Rowland in a recent poll, attracting 40.3% of votes.