Walk into Jimmy Kimmel's cluttered office and you'll see a space filled with a superstar showbiz nerd's knick knacks – from a portrait of famed comic Don Rickles to framed graphics that appeared on one of David Letterman's shows.

So it's not surprising that a guy who grew up worshiping late night TV legends has a tough time accepting that he's now an institution himself: the longest-tenured late night TV host currently on the air, with 20 years in the game.

ABC is celebrating the event with a special primetime episode of Jimmy Kimmel Live! Thursday featuring guests who appeared on the first show back in 2003: George Clooney, Snoop Dogg and Chris Martin from Coldplay.

The show has come a long way from its bratty early days when producers struggled to book guests and episodes had a raucous, frat party feel. Kimmel remains a little surprised by how it all turned out.

"I don't know how this happened...it's weird to be the oldest guy, but I guess I am," he said, laughing. "The institution I admired was David Letterman. [And] I never compare myself favorably to Dave. But it is interesting when people tell me, 'Yeah, I've been watching since I was eight.' It's very strange."

Excelling at a job he nearly quit

The day NPR visited, Kimmel hosted an episode featuring Rob Lowe, Storm Reid and Seal, cracking jokes with an unflappable, smart alecky flair. His skill at making improvised comedy bits work seamlessly is rooted in a confidence born from lots of experience.

These days, Kimmel's preparing to host the Oscars for a third time and balancing work as a producer of other TV series with Jimmy Kimmel Live! But Kimmel almost quit his late night job last year, hesitating before finally agreeing to stay on as host and executive producer for three more years.

ABC worked hard to woo him and he talked it over with his wife, Molly McNearney, who is co-head writer and an executive producer on the show.

"I was almost sure I was going to stop," said Kimmel, joking that he stayed in the job to avoid cleaning out his office. "Part of it is just OCD; 20 years just sounds right...Ultimately, I just couldn't decide what to do, and in those situations, I have a tendency to stay the course."

What drove him to nearly quit was the grind. Working with writers and producers to create a nightly late night show filled with guest interviews, pre-taped comedy bits and a long opening monologue requires generating lots of material. Which is a lot of work.

The show's rehearsal, held on the studio stage hours before the show's taping, offered a glimpse of the process. Dressed in a plaid shirt and pants – far more casual than the suit and tie he normally wears on camera — Kimmel sat behind his desk as staffers lounged in the seats usually occupied by the audience. (He was careful to tell a visiting reporter that they all clapped when he walked in the room as a joke, not because he asks for it.)

As video clips for his monologue played, Kimmel suggested edits and tossed off possible jokes. He said the process of writing and rewriting that continues all the time – on weekends, late at night, around the clock.

"It's relentless," the host said. "If you want to do a good job, you have to work all the time. You can't even watch television without thinking 'Is there something here that I can use on my show?' Because great ideas are hard to come by."

Showbiz success through hard work

The show's bandleader, Cleto Escobedo III, said Kimmel's commitment to such an intense work ethic is also why the show has lasted so long.

"He just works his a** off; he never stops working," says Escobedo, who grew up with Kimmel in Las Vegas and has led the show's backing band since the beginning. "I feel like he's earned every bit of success that he has."

Kimmel is surrounded by friends and family while working on the program. In addition to McNearney and Escobedo, Kimmel's uncle, Frank Potenza, appeared on the show playing a security guard until his death in 2011 (a small shrine to his memory sits at the studio entrance). His Cousin Sal and Aunt Chippy have also appeared on camera.

And there's his sidekick, a chubby, lovable teddy bear of a guy named Guillermo Rodriguez.

Rodriguez was a security guard at the studio's parking lot who slept inside the announcer's car during shifts (he was working multiple jobs and the guard shack didn't have a heater). Back in 2004, producers enlisted him to perform comedy bits for the show, like offering celebrities shots of tequila backstage at the Emmy awards and pestering players at the NBA playoffs media day.

But does it bother Rodriguez that the show often mines humor from his weight, heavy accent and drinking habits?

"Listen, it's comedy," said Rodriguez, adding that the income and fame from appearing on the program has changed his life. "So I laugh my way all the way to the bank. [Before joining Kimmel] I did so many jobs: I did construction. I did gardening. I did painting....On this job, they make fun of me, I make fun of them; I'm having fun, they're having fun."

When Kimmel was asked whether some of the jokes Rodriguez makes reinforce stereotypes about Latinos, he pushed back.

"A good way to murder comedy is thinking about big picture topics all the time," he said. "Guillermo is one of the funniest people I've ever met. I'm proud of putting him on the air. I don't know what I would do without him."

Kimmel said keeping this family of collaborators employed was another reason he decided to keep doing the job.

"A lot of these people I won't see much if I stop doing the show...when I stop doing the show," he said. "It all added up to [a decision to] stay."

A nervous debut from behind a desk

Kimmel wasn't expecting a 20-year tenure when Jimmy Kimmel Live! debuted on Jan. 26 2003. The show was preceded by a jokey announcement from Nightline anchor Ted Koppel: "There will be no special post Super Bowl edition of Nightline tonight, so that ABC can bring you...the following piece of garbage."

Kimmel, who had previously co-hosted programs like Win Ben Stein's Money and The Man Show, didn't seem the most likely choice for ABC's venture into late night talk TV. In the first episode, he's a bit heavier, more nervous and a little disheveled, wearing a suit without a tie and a white undershirt.

"Welcome to 'Enjoy It While It Lasts,' my new talk show," he cracked.

For a year, Jimmy Kimmel Live! was actually broadcast live, in an attempt to offer jokes that were topical as possible. And for 10 years, the show aired at or after midnight, behind ABC's newsmagazine Nightline, before the two shows switched timeslots.

"I spent the first couple of years sitting behind the desk [for the opening monologue], treating it like a radio show," said Kimmel. Then a network executive eventually asked him: Wouldn't you stand up to greet guests coming to your house?

"As soon as I did, I thought, 'Why didn't I do this sooner?' David Letterman stands up. Johnny Carson — everyone stands up. But I was scared. I was hiding behind the desk."

Over the years, Kimmel has developed a snarky, everyman style of comedy influenced by his early days as a radio producer and DJ, including elaborate pranks and a long-running, fake rivalry with movie star Matt Damon.

Then, in April 2017, his son Billy was born with heart problems and Kimmel teared up while criticizing a proposal by Donald Trump to cut nearly $6 billion from the National Institutes of Health. He said, "If your baby is going to die and it doesn't have to, it shouldn't matter how much money you make."

Later that year, he offered a similarly emotional, serious take on gun control following a mass shooting in his Las Vegas hometown. Perceptions about Kimmel changed. He was doing what his idols like Letterman and Carson had done before him – helping people make sense of a confusing world.

"A big part of what we do with this job is we put it in relatable terms," Kimmel said. "I think it's always comforting to hear that somebody else is going through something that you are going through...I hear from people who don't like me talking about Donald Trump [critically]. But I hear from a lot more people who do like to hear it, and who do appreciate somebody reminding them...that this is not normal and we don't necessarily need to accept it."

Yet he's had his controversies over the years, apologizing for wearing blackface during some sketches on The Man Show and apologizing to Abbott Elementary star Quinta Brunson for a joke where he laid on the floor during her acceptance speech for an award at last year's Emmys.

The end of late night?

These days, the classic role of late night TV shows – to sum up the day's events in a cheeky monologue right before the audience heads for bed – has been disrupted by online media.

Kimmel admitted that viewers often watch parts of late night shows online, which brings smaller ratings for the network (so far this TV season, Kimmel averages about 1.6 million viewers a night, behind Stephen Colbert's 2.8 million and slightly ahead of Jimmy Fallon's 1.5 million, according to Nielsen ratings provided by ABC). But even as big names like Trevor Noah and James Corden leave the late night genre, Kimmel has faith it will stick around.

At least, until he's ready to leave the job.

And when WILL he eventually depart late night TV? As a sort of answer, Kimmel recalls a joke he used to warm up audiences that reliably got big laughs, until it didn't.

"There was something about it that just stopped working," he said. "You always have that fear: 'When does it stop working?' It's going to stop working eventually. I want to be ahead of that. I want to stop before it stops."