The actor - who currently plays Andrew Cotton in EastEnders - explained that he could have ended up like his "villainous" stepfather.
"My stepfather, Terry, was a bit of a villain in the 1960s and 1970s. He was really well known and worked with all the main faces in the East End," he told The Sun.
"A lot of villains who grew up in those times felt like victims of society. Because I couldn't read and had all these teachers calling me thick, I could have thought, 'I'm going to show you how thick I am'.
"I could have easily gone in that direction. I thought about it. But thankfully I didn't."
Grover explained that he chooses to turn any anger issues into positivity, and has always lashed out against "bullies".
"I was a bully's bully. I always detested bullies. I would get into fights protecting people. You have some hairy situations on the streets of the East End, but you get used to it."
The actor added that he fondly remembers growing up around London gangsters as a child.
"Although there were a lot of villains, there were lots of principles and morals around. People would never buy anything that was nicked out of someone's house, for instance. They'd only ever buy anything nicked off a big corporation because it felt morally right.
"It was good growing up in that environment. Everyone knew each other and there was respect. I think they were, in some ways, the good old days."
Ricky Grover recently revealed that his showbusiness career was saved by the late Bob Monkhouse.