-- By Eilene Zimmerman, Stanford Business Contributor

Invention has another mother: failure. It may seem counterintuitive, but repeated failures can, and often do, lead to success. Every time we try something new and fail, it provides valuable information about what went wrong and, as important, what went right. From that, we can make small changes and try again, continually learning and innovating.

"If you're trying to solve a problem, there are potentially hundreds of possible pathways to take, but only a few are going to lead to the appropriate solution," says Baba Shiv, a professor at Stanford Graduate School of Business whose research focuses on innovation. "And the only way to discover that is to try and fail and try again."

Experimentation and failure are essential to innovation because, by its nature, an innovation is an unknown that can only be discovered through trial and error. Still, for all the startups that follow the mantra of "fail fast," there are many corporate leaders who see failure as something to be avoided, not embraced. Shiv has categorized this fear-of-failure mindset as Type 1. An innovative point of view, one that perceives failure as exciting because of the opportunities it presents, he labels a Type 2 mindset. "For Type 2 people, the challenge is to keep experimenting and learning until they get to what works," says Shiv.

In corporate hierarchies there is a tendency to give greater weight to the opinions of leaders rather than their subordinates. However, those opinions are usually based on instinct rather than information. The one thing that can trump a higher-up's opinion is data, and repeated experimentation and failure lead to a lot of it, says Shiv.

Data can also win over the opposition. Those with a risk-averse mindset generally oppose innovating through experimentation, like rapid prototyping or continual iteration. "They are constantly looking for ways to mitigate risk, because that gives them comfort and reassurance," says Shiv. "Those are things you can also get from data."

Yet repeated failure can be tough to justify to management because of the money and time--yours, your team members', your manager's--involved. Experimentation often takes resources away from other areas of the organization that need them, which is why managers feel obliged to see results, although there may be nothing to show yet for the work. "Employees have to justify the investment being made now, even though they don't know if they will have anything to show for it and if they do, it will be in the future," says Shiv.

Organizations also tend to reward big breakthrough successes rather than smaller ones, but those game-changing innovations generally happen after, or in tandem with, the incremental ones. Shiv uses the example of Toyota, a company that encourages experimentation across its organization. "Most of those experiments result in incremental changes like improvements in production and manufacturing," he says. "At the same time, however, the company invested in hybrid technology."

Shiv says this two-pronged approach relies on exploitation and exploration. Both, he says, are essential for real success. Exploitation for Toyota means honing competencies it already has to reduce costs and improve the value for customers--incremental innovations. At the same time Toyota invests in exploration, which enables breakthrough innovations.

"If you don't invest in exploration, someone else will, and then you'll just be licensing or acquiring their know-how," says Shiv.

The process of exploration also builds a large knowledge base for a company, information that couldn't be gained through an acquisition. Yet many companies are hard-pressed to justify a focus on exploration in a world of increasing short-term pressures, especially on public companies with stock prices to worry about and dividends to pay. Those pressures cause businesses to be more oriented toward exploitation and the small innovations that come with that, rather than exploration and its bigger risks and rewards.

When companies do support experimentation, Shiv recommends they test products or services on the average customer, not highly knowledgeable customers or early adopters. "The average customer is far less forgiving," he says. One thing all companies can and should do is encourage a culture of innovation at all levels, the way Google has. "Small teams within Google run 3,000 to 5,000 experiments a year," says Shiv. "A manager there might say, 'We should use this course of action,' and a new hire might say, 'I think this way of doing it will be better.' That manager's first reaction isn't resistance, it's 'OK, let's test it.' That's the great thing about Google. There are no egos when it comes to opinions," he says. "There's data."

This article is from Stanford Business Insights. Follow us on Twitter.