In the early days of Late Night With David Letterman, the host and his then-girlfriend Merrill Markoe would sit through meetings with NBC executives, trying to figure out what sort of sketches they could run in their new 12:30 a.m. slot. The rules, according to a 2011 Rolling Stone story, were established so his show wouldn't infringe on Johnny Carson's territory: He couldn't have a sidekick (like Ed McMahon), he couldn't have a horn section (like Doc Severinsen's), and he couldn't bring on comedians like Don Rickles and Bob Newhart, who were staples on Carson's show.

As for the oddball segments left over from Letterman's canceled morning program: The newly invented "Stupid Pet Tricks" could stay, but Letterman would have to tone down the bits to be more mainstream. "You mean," Markoe asked the suits during one meeting, referring to a nearly 100-year-old sideshow trick, "do 'Stupid Pet Tricks' using, like, a horse that can count?"

"Exactly!" the NBC execs replied.

And so it was pretty much decided--whatever the network brass said, Letterman and Markoe would go the other way.

As Letterman steps away from his post and heads off to Westchester or Montana or wherever it is he chooses to disappear to, TV is losing more than its reigning late night king. It's losing an underdog who rose from a small town in Indiana, failed repeatedly, and ended up on top, even if the numbers almost never showed it. And it's losing someone that anyone who hopes to be a rebel or an innovator should admire.

On paper, Letterman's earliest days on TV were a series of failures: His spot as a cast member on Mary Tyler Moore's variety show disappeared, along with the show itself, after three episodes; he hosted a pilot for a game show, only to see it get canceled before ever airing; the 10 a.m. David Letterman Show survived just over four months before being pulled.

But the underdog comedian persisted. He caught the eye of his hero, Carson, thus catching his big break--he earned a few spots on the legend's show, and then landed Late Night, which premiered in 1982.

Even once he got that break, he refused to do things the way they were done before he arrived. While other hosts were holding standard showbiz Q&As, Letterman was catching his guests off-guard with post-interview interviews. Back when it was still customary to host a show from one's own studio, Letterman was often on the street, using his relative obscurity as cover while he baffled storeowners and pedestrians. He turned the late night format upside down.

It's not a stretch to say that, for better or worse, he paved the way for a large chunk of what's on TV today: from Jackass-style stunts and undercover comedy to Billy on the Street's ground-level absurdism. He also cranked out hits that didn't even yet have a place: strange, viral-worthy segments of Dave dropping things off tall buildings and dunking himself into tanks of water wearing an Alka-Seltzer suit pre-dated YouTube by 20 years.

Yet as far as ratings go, Letterman rarely held the No. 1 late night slot throughout his 33-year run. After a decade in Carson's shadow, he regularly finished second to Jay Leno's easy-to-digest style. In a way, though, this feels right. No. 1 would just be too mainstream for a person who, in what's probably a quiet act of rebellion, wears clashing light gray socks under his dark suits every night.

Letterman made his career defying the status quo--even if the status quo was something as simple as "not featuring bowling dogs on talk shows." He innovated, sometimes out of necessity, but often because his own brand of humor didn't yet exist in American pop culture. Whether you find him funny or not, there's no denying that he changed the game for everyone who came after him.

There might not be another King of Late Night after May 20. But there will be plenty of rebels, on TV and otherwise, who rise from anonymity and leave their mark. Many of them will have David Letterman to thank.