How well do you know your job? Really, really well? Because you've been doing it for a very long time? Congratulations. You're likelier than most to make the same mistake that's plagued everyone from financiers to governments to sports team managers: Relying more on your own judgment and experience than the data.
That observation comes from Michael Lewis, bestselling author of Moneyball and many other books. Lewis has built his career on telling stories of people who outsmarted their competitors by following the data instead of their own instincts. In Moneyball, Oakland A's manager Billy Beane recognized that other, better-funded teams were over-valuing some statistics such as batting average, and undervaluing others such as on-base percentage (which includes the unspectacular but useful ability to get oneself walked). Beane used that knowledge to get to the playoffs and compete with teams like the New York Yankees that spent up to three times more on their players' salaries. In The Big Short, Michael Burry followed the data and figured out in 2003 that the housing bubble would burst in 2007 and take some of Wall Street's most venerable firms with it. While the rest of the market collapsed, his investors profited by 489 percent a year.
Lewis' latest book is The Undoing Project, which tells the story of two Israeli psychologists who studied how humans make decisions, discovering that we're all a lot less rational than we think we are. He discussed that book, and what he's learned about human decision-making, at the recent Insight Summit produced by survey software company Qualtrics in an onstage interview with Qualtrics CEO Ryan Smith.
The hunger for an expert who can give us certain answers in an uncertain world is always with us, Lewis explained. "Even in the most data-drenched operation, this question of when to let human judgment back in is always there." But why is data better than human judgment? Well, for a few reasons:
1. Data doesn't look for patterns where there are none.
This is a well-known cognitive failing of the human brain--we are always looking for patterns and connections. Sometimes we see real patterns, for instance, when we observe that people who smoke cigarettes have a greater likelihood of getting lung cancer. Sometimes we see patterns that really aren't there, such as the "zero curse" in which every president elected in a year ending zero supposedly must die in office. Up until 1980, that was true, and though Ronald Reagan did not die until many years after being president, he narrowly avoided dying when John Hinckley shot him in 1981. But then, of course, George W. Bush broke the pattern completely. Elected in 2000, he's still alive and kicking.
In sports, this desire to find patterns leads team managers to link players they're evaluating with players whose careers are already known and bear them some physical or other resemblance. But Lewis says you can overcome that tendency with the following rule: Do not compare the player you're considering with another player of the same race. "Once you force people not to use race, they cease to see those connections," he said.
2. Data can't be influenced by physical appearance.
Lewis said this realization struck him the first time he was in the Oakland A's locker room and watched the team undressing. "They didn't look like professional athletes," he said. Ungainly, overweight, or clumsy looking it was easy to see why managers from other teams had looked at these players and not seem potential stars. But data is blind to things like a pot belly or an unwieldy throwing style, and it was able to see the truth.
3. Data has no ego.
If you really dislike reading how data is smarter than humans, it's probably because--like most humans--you have an easily bruised ego. If your skills and mine are less useful than that of a computer...well, where does that leave us in a competitive job market?
This is why, even when faced with overwhelming evidence that they should trust data, human experts so often turn away, relying on their own "priceless" instincts instead. In the introduction to The Undoing Project, Lewis recounts a spectacular instance of this kind of myopia by the Boston Red Sox. After Moneyball came out, the Red Sox started using a data-based approach to hiring, similar to the Oakland A's that Lewis described in the book. In 2004, the year after the book was published, they won the World Series. They won again in 2007 and 2013. But then, after three bad years, the Red Sox announced in 2016 that they were abandoning the new system, with owner John Henry noting that, "We have perhaps overly relied on numbers..."
And so, they have gone back to their old system of relying on human expertise. The last time that system won them the World Series was 1918.