Simon Cowell feels like a father figure to a certain One Direction member

Simon Cowellput together mega-selling British band One Direction after the boys auditioned separately on X Factor in 2010. Though Harry Styles, Zayn Malik, Liam Payne, Niall Horan and Louis Tomlinson parted ways in 2015 — Cowell, 5 still feels especially fatherly toward one 1D alum in particular.

“[Louis] was like 17 when he auditioned for me,” the America’s Got Talent judge told Us Weekly and other reporters at a taping of the NBC show on Tuesday, August 14. “And now he’s a judge for me on [X Factor in the U.K.] and he’s got a son. We have gone through so much together but the fact that we’re still friends is fantastic. I like that.” (Tomlinson, 26, shares 2-year-old Freddie with ex Briana Jungwirth.)

Though the group has no plans to get back together, Cowell thinks they should reconsider. “Personally, I think they would enjoy it more the second time around because they haven’t got anything to prove,” the producer explained. “They’ve done it, they know there would be a demand for it, and I think they would have more.”

See Louis Tomlinson with his mother through the years:

As for if they never reunited? “Then I would be very unhappy,” Cowell confessed. “With the amount of hits they’ve had, and because they’ve had time away, I think they’d probably enjoy more the second time around. I just have that feeling.”

While Cowell feels paternal towards Tomlinson, he is the real dad of Eric, 4. Since welcoming the little boy with girlfriend Lauren Silverman, Cowell admitted he’s softened up — especially when it comes to contestants on America’s Got Talent. “Because you kind of think, ‘What would I feel if that was Eric up there?’ Well first of all, I wouldn’t let him. I want him on my side of the desk, not the other side,” Cowell quipped. “But yeah, you definitely feel differently. I mean certainly for the younger contestants.”

He added: "I’m hoping he will do what I am doing and work with me. I think that would be my dream.”

America’s Got Talent airs on NBC Tuesdays at 8 p.m. ET.

See Simon Cowell through the years: