The 41-year-old wannabe singer, who made it to the final three on the reality television series earlier this week, said that he sometimes feels isolated because he is the only dad taking part in the competition.
Wishart told New Idea magazine: "In the competition, everyone is missing someone, but I guess I'm the only one with kids... I have to say, I miss them all terribly."
He said that he also struggles being away from wife Robbie, who he grew up with.
"We've always been in each other's lives," he added. "We went to [kindergarten] together, then primary school. It's crazy when you think about it."
Robbie said: "I miss having Andrew to cuddle. At the moment, [14-year-old Tully, 12-year-old Levii, 11-year-old Bahlin and 8-year-old Indiah] are in with me every night.
"It's actually painful watching [the show]. It's definitely been challenging."
Wishart is mentored by judge Natalie Bassingthwaighte in The X Factor, while his fellow finalists Reece Mastin and Johnny Ruffo are both looked after by Guy Sebastian.
Ronan Keating's last remaining act Three Wishez were eliminated earlier this week, while Mel B's last protégée Christina Parie left two weeks before.
> Reece Mastin 'doesn't want to be Aussie 'X Factor' favourite
> Three Wishez blame female fans for Aussie 'X Factor' elimination
Watch Andrew Wishart performing on The X Factor below: