From a rational point of view, starting a business is crazy: The failure rate is high, the emotional toll is high, and the likelihood of having to work extremely hard for potentially little reward is incredibly high.

But every year millions of people take the entrepreneurial plunge, and many succeed. Why do some succeed while others don't?

Here's another in my series where I pick a topic and connect with someone a lot smarter than me. (There's a list of some previous installments at the end of this article.)

This time I've gone straight to the top, to a guy who has spent much of his professional life interviewing, studying, writing about, and celebrating the entrepreneurial spirit: Eric Schurenberg, editor-in-chief of Inc. (And yes, ultimately he's my boss.)

In very broad strokes, why do some entrepreneurs succeed?
In 2003 the economist Amar Bhide wrote "The Origin and Evolution of New Businesses." He looked at founders on the Inc. 500 to determine how they were different from the rest of the people who start businesses.

He found the people who reached that level of success were in many ways no different: They didn't start with a comprehensive business plan, didn't have a deep knowledge of their industry, didn't have a competitive advantage like special access to financing. So he came to two conclusions.

One was that the entrepreneurs who succeed tend to be in a field where tremendous uncertainty exists, so they're rewarded for taking significant risk. That's why technology is such fertile ground: plenty of uncertainty, change takes place at a rapid pace, incumbents often disappear quickly, the barriers to entry are low.

It's easy to start a software company in the basement, but not an automobile company.
Absolutely. Plus the Internet affords tremendous potential for scale. So, again in very broad terms, one set of entrepreneurs surfs the waters where conditions change quickly and they manage to catch a maverick wave--or they wind up becalmed or squashed by a rogue wave.

Tech entrepreneurs certainly gain a lot of attention, but for every tech success there are dozens of successful stories that aren't based on "new."

True. That's why Bhide also noticed the Inc. 500 tended to be overly populated by service companies. The key in service is differentiation, often on a personal level: You might not have a built-in advantage like financing or specialized knowledge, but by dint of your personality and hard work and chutzpah you can still succeed.

You can be like Gary Vaynerchuk and return every single email you receive. Or you can be incredibly outgoing or work impossible hours or give up your firstborn to make that sale--those people can gain an advantage that can translate into success simply by the force of their own personality.

What else have you noticed from analyzing the Inc. 500?
One story nearly every entrepreneur shares is that moment of near death: almost bankrupt, down to the last dollar, credit cards maxed out, house approaching foreclosure... and then a customer shows up. Or they land funding. Those are such incredible stories, but the ones we hear about are ones where the cavalry rides to the rescue in time.

We tend not to hear the stories where the cavalry never shows up, and the entrepreneur licks his or her wounds... and then starts another company.

I've always thought the failure rate should scare more people away, if only because for years it scared me away.
People tend to underestimate the risks of entry and failure, at least on a personal level. There's this famous survey where people were asked about their driving ability and 80 percent said they were above average, even though the respondents were all people who were injured in car accidents.

Entrepreneurs are particularly prone to overconfidence--and they need to be.

I like to call that irrational optimism. To succeed at a high level--in business, in sports, in anything--you have to push aside all the doubts: feelings that you aren't smart enough, dedicated enough, adaptable enough... or that in spite of your best efforts you won't succeed.
Absolutely. What person with a ton of talent and labor value would throw that away to pursue a venture with a low likelihood of success and a high risk of failure, both financially and personally?

From an economist's points of view entrepreneurship is a total conundrum: It shouldn't happen in a world where people make rational decisions to advance their best interest.

Yet entrepreneurs succeed every day--many of them beyond their irrationally optimistic dreams. That's what makes entrepreneurs so fascinating.

So let's talk about failure. Why do many entrepreneurs fail?
In broad strokes, tech companies tend to fail because there's only so much funding to go around. The cost of getting on a customer's radar screen is often high--switching costs are sometimes too high even if your product is better. In essence, many companies fail simply because it's so hard to succeed.

In service businesses there are often no inherent advantages to the customer. We won't switch to another provider unless we're totally blown away by the personality of the entrepreneur.

My theory, based on zero science, is that most entrepreneurs have one skill that brings them to the dance. Your one skill gains you entry, but when you try to grow past a certain level your lack of other skills is a killer. Lots of entrepreneurs I talk to say what got them past that near-death moment was finding a partner with complementary skills.
There are so many institutional and systematic pitfalls when you try to scale past initial success. And don't forget many start-ups end up in that funding no-man's land: too small for venture capital but too big for ever-increasing resource needs.

Managing growth is hard. Ultimately you don't just need a partner who complements your skills. You need a bunch of "partners": a great CFO, a great CTO, any number of great people who bring specialized skills. Without them, growth is often impossible.

I also think a lot of people overestimate their "general" business skills--what works in one industry doesn't necessarily translate to others. People like Norm Brodsky, whose business seems to be business, are somewhat rare.
A good way to interpret Norm's success is to say he has a talent for spotting opportunity and the skills to implement successfully. Take City Storage: He owned a bike messenger service and a customer needed a place to store documents--so he started by storing boxes in his office. Then he used his business acumen to build from there.

What does this year's Inc. 5000 say about entrepreneurial success?
Like in many other years but to a somewhat greater degree, a significant percentage of the 2013 Inc. 5000 are B2B service companies: terrific salespeople and plenty of elbow grease. Don't forget B2B success is not necessarily easier but it's different than B2C: Land a couple of big contracts and you have a going business. When you own a sandwich shop you need to attract a ton of customers to survive.

Take the No. 1 company on this year's list, Fuhu.  Fuh's founders embody everything we've discussed. This is the seventh company its president, Robb Fujioka, has started.

Fuhu didn't invent anything totally new: They found a niche by creating an inexpensive Android tablet for children. They rode uncertainty to catch the wave of tablet adoption.

The problem with tech is that you always need a "next."
Fuhu is already working on a number of "nexts," and that's something else successful entrepreneurs have in common: Next is in their DNA.

Check out other articles in this series: