It's fascinating to watch people get excited about "big data" or, as it was once called, "data." Cross-referencing data-sets from different places is a powerful tool, but it's also dangerous in two ways.
1. Big Data Bad Question = Dumb Strategy
As I explained in my post about "owning the question," every question contains an implicit "frame" that predefines the answer you'll get. That's certainly true when asking questions of humans and doubly true when asking questions of data.
If the question that you ask of data is fundamentally flawed, you'll get an answer that's worse than useless. It's not just "garbage in, garbage out." It's "garbage question in, toxic waste out."
I could speculate about some companies (Facebook and Microsoft come to mind) where I think this mistaken use of big data is happening today, but it would take years before we'd know for sure.
So instead, here's an example from the past in which the outcome already known.
In the early 1990s, the high-tech company that had the closest equivalent to "big data" was the minicomputer manufacturer DEC.
DEC had its own internal version of the Internet complete with social networking, a complex supply-chain database, a detailed database of customer purchases and service requests, and online access to hundreds of market research reports.
In 1991, I was present at a meeting during which a marketing executive presented the results of an extensive search and analysis of all this data. His conclusion was: The PC is a fad and the computer industry would soon return to using character-cell terminals.
This prediction was ludicrous to the point of insanity, but it was taken seriously because the data really did prove that, if you asked something like: "What is the sales growth in terminals inside our 10 largest customers over the past five years?"
Such a question presupposes the answer and therefore the corporate strategy that DEC subsequently and disastrously pursued into ruin. The question reflected not the potentialities of the data, but the peculiarities of DEC's PC-phobic corporate culture.
As a result, a question that might have changed the company's direction, like "How much money is being spent worldwide on minicomputers compared to PCs?" never got asked, even though the data to answer that question was readily available.
If you're using big data, you need to be careful that the questions you ask of it don't reflect your own biases and views of the market and the world. If not, you'll inevitably end up with a skewed perception of reality.
2. Big Data Misses Black Swans
Big Data is excellent at reflecting the past and present with great precision and detail. It is thus highly useful for targeting your marketing and sales and product development and for determining, through analysis, which strategies and tactics are likely to work in the future.
Assuming you're asking the right questions, that's all to the good. But there's one thing that Big Data is terrible at: predicting and identifying "black swans"--those surprising and sudden events that utterly change the rules of the game.
It's unlikely that any analysis of the Big Data available in 2008 could have predicted the rapid success of the iPhone and the sudden crash of the financial industry. Quite the contrary. Pundits with access to "big data" didn't see these coming at all.
The same can happen in smaller companies, too. The entire concept of "disruptive innovation" assumes that there will be "black swan" technology or concepts that come out of nowhere and change an entire industry.
Big Data can't really tell you when that's going to happen.
Please don't think I'm saying that Big Data is a bad idea. However, I do think it's a mistake to depend too heavily upon it. Ultimately, to be successful, you need objectivity, creativity, and intuition, none of which you'll ever get from data, regardless of size.