You've likely heard so much about Big Data this year that it's become a permanent fixture in your corporate vocabulary. As if on cue as the year winds to a close, now a consultancy has declared 2014 as the year of "small data."
I know--who would have thought that a business would unilaterally declare its own annual theme in an area that it just so happens to be pushing? But don't write off the concept because it makes a great deal of sense. And I don't say it because I've found myself independently using the term "small data" while looking at the practice of data-driven decision-making and operations.
Big Data Is Tough Data
The consultancy mentioned, the Digital Clarity Group, has pointed out a major problem with big data: Although the promise of uncovering critical hidden patterns is alluring, making big data work for your business can be devilishly difficult:
Yet all the steam coming out of the big data hype machine seems to be obscuring our view of the big picture: in many cases big data is overkill. And in most cases big data is useful only if we (those of us who aren't data scientists) can do something with it in our everyday jobs.
In other words, to really make use of big data, you need the following things:
- lots of computing horsepower
- infrastructures that can handle the types of processing necessary
- the right tools to make it work
- people who not only really understand statistics, but mathematical modeling
- data clean enough that its state won't invalidate your results
- enough time and resources to go fishing
- sophistication about the methods to know you're not getting a quick breakthrough
Some companies pull it off admirably, as you should notice every time an ad just happens to reach you at a time you'd consider such a product or service. But even most large companies have problems making big data work for them. Startups--you know who you are--rarely have the resources to throw at using big data, unless that's actually the business they're in.
Instead, you can focus on the data you already have, or that is readily available, and that is directly applicable to your business. Then you deliver analyses and results that people in your company can actually use now.
As the Digital Clarity report puts it, you have transactional data, online data that can give insight into customers, and social data that is available to provide various insights through text and sentiment analyses.
Master What's at Hand
It may be that aspects of small data--like social network information--are actually the digested results of big data that a third party can deliver to you. That's fine. But it has to be something you can readily use. One example that landed in my email today was a company called Skymosity. Frankly, the pitch and an online video were too self-consciously hip and ironic, taking far too long to get to the point. But the ultimate point was simple and compelling: a company can learn how various weather patterns affect sales by ZIP code and then use this information to create highly targeted email campaigns based on weather forecasts.
Small data can be that clever. It can also be something more basic, like understanding what products your clients ultimately tend to buy together or if there are demographic patterns in your sales.
Don't get distracted by the big data that you're supposed to be chasing. If you can handle it, that's fine, but only after you've mastered the small data that you already have.