Reginald VelJohnson Once Walked Out on 'Family Matters' over the 'Challenge' of Working with Teens

Family Matters star Reginald VelJohnson has a bit of personal experience with that old showbiz adage "never work with children or animals."

VelJohnson, who played Carl Winslow on the ABC series from 1989 to 1997, opened up on Tuesday about his frustration — and fondness — for his young costars.

Most notably, the breakout character from the TGIF sitcom was Jaleel White's Steve Urkel — and though viewers found the archetypal TV nerd hilarious, VelJohnson admitted he was not always amused by White off camera.

"He was a wonderful kid, [but] I think working with him was a challenge," the actor, 69, told Entertainment Tonight.

He acknowledged, "Working with everybody was a challenge, and there were some moments where he was a little difficult. But overall, I have nothing but good memories from him."

VelJohnson noted, "I think, you know, a lot of the kids grew up on the set and we had to deal with children ... and that was an interesting experience, dealing with children as they grew up."

Sometimes, the kid actors' antic even drove him into hiding: "I went to my dressing room and locked the door when I [rehearsed] my scenes because that was the only way. Because kids have a lot of energy ... [and] you couldn't deal with it."

On another occasion, he recalled, "[Jaleel and] the kids were playing basketball on the set, and I just couldn't take it. I said, 'Please, please stop the basketball!' And they wouldn't stop."

VelJohnson continued, "I said, 'Listen, you either stop the basketball playing or I'm gonna leave.' And I left. But there was no more basketball after that."

But the Die Hard actor had nothing but kind words to say about JoMarie Payton, who played his onscreen wife Harriette.

"I just fell in love with her the first time I saw her, I really did," he said.

He continued, "I think that she's a special lady, you know, and that's how I approached the role. Because she was the best to work with."

"We never had arguments or anything on the set. he added. Never. Not once, not ever," he said. "I really appreciate that because I made sure that our relationship on the set and off the set was gonna be a good one."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

All nine seasons of Family Matters can be streamed now on HBO Max.