It appears that big data is everywhere. My iPhone tracks my steps whether or not I activate the "Health" app. The Internet of Things allows for the synchronization of my Easy Pass and my bank account. And companies continue to hope that they will develop the perfect algorithm to be the code cracker illuminating just what the coveted millennial desires.
But here is the thing: big data alone will reveal only a portion of the full picture. Big data gives us a bird's eye view: it points out patterns in the field. These patterns are indicative of something- but what is the thing driving those patterns?
To understand that, we also need "deep data", or what I like to call "the worm's eye view", complementing big data's bird's eye view, in order to understand the catalyst stirring up all of the detailed activity we discern from the meta level. The worm's eye view is derived from qualitative research: data that comes in the form of conversations, interviews, stories and observations.
If your firm invests only in big data as the sole source of enlightenment about markets, then you will have limited results, because you are only getting 1 layer of the multidimensional and dynamic reality of markets. The real insights are generated from an integrative process, one that values both methods. I'd bank on hiring anthropologists, designers, storytellers and strategists with a design thinking competency. Those companies that begin to reframe who they hire will net results which turn a profit in the longer term.