4 Lessons From Nate Silver's Election Predictions
BY Erik Sherman
Why do big election predictions matter? They offer key lessons about how to be smart with 'big' data.
Unless you've been safely ensconced in a media-free hideaway, you've probably heard of Nate Silver, the predictive phenom who writes the FiveThirtyEight blog for the New York Times. Controversy has flourished because of the now 86% odds he gives Obama for winning the presidential election.
A lot of people, including Times conservative columnist David Brooks and MSNBC personality and former politician Joe Scarborough, disliked Silver's air of certainty about what they think is an uncertain process, and said so loudly. Then, of course, came the defense of Silver in other parts of the media.
Put politics aside for the moment. There are some important lessons in this discussion for entrepreneurs who are interested in what you could call "big data." As in predicting elections, in running a business it's easy to follow the path of many partisans and pundits and make fundamental mistakes about how to add up the distribution of risk and spot legitimate opportunities.
Lesson # 1: Prejudice doesn't trump prediction.
The fundamental issue around the current predictive controversy is that critics largely misunderstand what Silver is doing. They might want to believe that you can't use math to predict human action. But I disagree: At any time, you may be able to see trends that give you an insight into what will happen.
The book Moneyball, which chronicles one baseball club's success in using statistics to cost-effectively run a team, is a perfect example. The talent market in the game was hugely inefficient. Teams put premiums on talents or aspects that satisfied managerial assumptions and prejudices, but that didn't necessarily result in better performance. At the same time, capabilities that showed a statistical impact on the game over time were undervalued.
Don't assume that there are no hidden patterns in your industry or business. What do you think the whole "big data" concept is about?
Lesson #2: Probabilities are relative.
Silver's critics say that polls aren't predictive because they represent one point in time. That is correct. However, when you gather data over enough time, the results may show a direction that is likely. That's what can make the difference to a business.
Being exactly right all the time is overrated. Talk to a scientist about quantum mechanics and you'll learn that what we see as reality is the accumulation of many probabilities, and yet the results are substantial.
What you need to learn is how to evaluate probabilities reasonably. One of the criticisms of Silver is that he never gave Romney much more than a 40% chance of winning--and, remember, we're talking about winning enough total electoral votes, decided on a state-by-state basis, not winning the apparently close national race. And a 40% chance is just under the probability of getting a single pair when you're dealt a poker hand.
Similarly, even an 86% chance doesn't mean a lock-in. When the odds seem heavily in your favor, you can still lose. That's why odds work best over long periods of time with the recognition that you win more often than you don't.
Lesson # 3: Patterns can add up in substantial ways.
What upsets so many about Silver's prediction is that it seems to fly in the face of national polls that put the election at an almost dead heat. That is a combination of wishful thinking--the pundits almost always make their coin on prognostication--and a simple but profound misunderstanding.
Not all things will affect a total system, like a nation or a business, equally. In the case of presidential politics, there are two races. The popular race, which is neck-and-neck, doesn't change the outcome of the election. The winning ticket is getting the right combination of individual states.
On some level, the pundits know and admit this. But there's a difference between acknowledging a fact and actually embracing its implications. Look at the financial industry, which has been talking forever about once-in-a-century so-called black swan events, but which seems to have a major meltdown every decade. Why? Because if you have 10 different once-in-a-century problems that can happen and they can happen independently of each other, then you've just gone from one time in 100 years to one time every 10 years.
Lesson #4: You're never too small to make data work.
Nate Silver may be gifted in predicting odds on events, but he's also just one guy. If you're not an expert, find someone who is and get the advantage to be had from knowing more about the future than your competitors.