Humans are a limited species. We cannot, with any certainty given the dynamic nature of life, predict the future.
In some cases you have better odds when running models and scenarios based on judging the future by analyzing past behaviors and patterns, such as insurance actuary tables or credit scores; however, most predictions are wrong.
The reasons are many, but the primary fault factor is that data is always limited and lacking context, and yet, in the world of business and non-profits we act as if it is holy. For example, how many times have you heard, "Where is the data to support that?" or "Is that just your opinion or is that statement backed with real data?"
Yet, an over-reliance on data and an underlying and unacknowledged fear of failure makes us give data a higher role than it deserves. This anxiety drives many middle managers to try and use data to predict the future, especially in the murky, grey areas of innovation, new channel development, and branding where markets and the people that make up the markets are psychological rather than logical.
A clear case of Innovator's Dilemma perplexes many organizations. They yearn to launch something new, but because it is new there is no data (no benchmarks, no analogues), so they lose courage and pass on the opportunity to crank out yet more of the same.
Data is one of the Meta-Games we play. Whether they are in politics, sports, or markets, we take pleasure participating in dataplay. Yet, most models trying to predict the future are wrong.
And guess what?
They are all based on data. Let's look at some proof-based evidence in politics, market, and sports predictions.
Look no further than the Pew Research Foundation's headline about the 2016 U.S. Presidential Polls: "Why 2016 election polls missed their mark."
According to the Telegraph: "American statistics and polling analysis site FiveThirtyEight gave Trump a 30 per cent chance of victory going into the final few days. This chance - the same as winning at rock, paper, and scissors - was generous compared to other forecasters."
As for money markets, take the great Forbes headline: "It's Official! Gurus Can't Accurately Predict Markets."
Then there are sports where the professional data analysts fare no better than amateurs with the fantasy leagues. Here's how well the biggest services - Yahoo, ESPN and CBS - translated their player stats into player rankings. After the first nine weeks of the 2010 N.F.L. season, here's how each site's projections ranked in Overall Accuracy (combines weighted scores for the QB, RB, WR, & TE positions): Yahoo Sports - #18, CBS Sports - #24, ESPN - #39.
My point is simple. All of this reliance on data gets in the way of market testing to see how well something new actually works. Data provides an excuse not to take risks, but is mostly unreliable, save for a few fields.