Sources claim that the pair embarked on a fling in 1984 while Jermaine was still with first wife Hazel Gordy, but that the romance was broken off when he refused to end his marriage.
"Her death has devastated Jermaine because of the history between them," an apparent insider told The Sun On Sunday. "No one understood the scale of his grief because they didn't know what they shared."
Jermaine paid tribute to Houston on Twitter last month, writing: "The right words fail me to sum up the sense of loss following the passing of Whitney... a wonderful woman, a true heart, and an immense artist.
"Now is not the time for anyone to indulge in negative speculation. It is a time to honour her talent and a voice which the world enjoyed. She is gone too soon, but that incredible voice will live on as an eternal reminder of her timeless talent. Rest in peace my friend."
Houston is believed to have based the video for her 1985 single 'Saving All My Love For You' on the romance, while Jermaine appeared to hint at their heartbreak in his memoir You are Not Alone: Michael, Through A Brother's Eyes.
Discussing his close friendship with Houston, he wrote in the book: "As much as I wanted to lose myself in all these feelings, I told her to wait. Ultimately, we had to go our separate ways and it killed us both."
Jermaine also wrote that Michael had warned him off an affair, saying: "He reminded me about Hazel, about family."
The 57-year-old remained good friends with Houston, seeking comfort from her during his divorce from Gordy in 1988 and the passing of Michael in 2009.
Jermaine married Halima Rashid in 2004.
Michael's friend David Gest recently claimed that the 'Thriller' singer and Houston were once "very much in love".
"I think she really wanted to marry him but, although he had a crush on her, he was too shy," Gest alleged. "Later, he confessed to me he should have made every move.
"If they had got together, I believe Whitney would not have done drugs or become a semi-recluse."