Steve Lohr has written a book about how to gain insights from massive amounts of data. I read it in about two days, which is a record for me when it comes to business books. I wanted to find out more after reading it, and asked him to elaborate. Take it away Steve.
From Steve Lohr, author of Data-ism:
The data deluge just keeps rising, doubling in volume every two years, mainly because of all the new digital data streaming from the web, social networks, sensors and smartphones. But there's a lot of water in the ocean too; you can't drink it. For companies, the goal is to distill insights, if not wisdom, from the data. In that quest, here are four things to keep in mind.
1. Measure what counts.
Jeff Rothschild, a Silicon Valley veteran who joined Facebook not long after it was founded, says a big part of the social network's success was a measurement insight. The conventional measures of a Web site's success are page views, ad impressions and the amount of time spent on a site by each visitor (engagement).
But Facebook closely monitored the flow of information among its members--posts, status updates, news articles sent to friends, or any acts of creating or sharing information rather than just being consumers of information. Investment decisions and product designs were made accordingly.
"It's not about the size of the data," Rothschild says, but about using data to "steer the organization based on the metrics you value most in the long run."
2. Start with the customer.
Accenture surveyed more than 1,000 companies last year, and asked them, "Where will big data have the biggest impact on your organization in the next five years?" The No. 1 response, by a large margin, was the impact on "customer relationships." Sometimes the crowd is right.
The new world of data opens the door to a deeper and broader understanding of customers. New data-driven lending start-ups, like Affirm, Earnest and ZestFinance, are using data to assess the creditworthiness of borrowers. The predictive picture is compiled by collecting an array of information from diverse sources, including household, buying habits, bill-paying records and social network connections. It amounts to a digital-age spin on the most basic principle of banking: Know your customer.
IBM in a partnership with Twitter has begun mining the social network's 6,000 tweets per second, and its archive of 140 character-or-less messages going back to Twitter's origins in 2006. IBM is using its Watson artificial intelligence technology to scrutinize the Twitter information fire hose, and offer results via the cloud as "insight as a service."
Not just big companies, like IBM, but start-ups, like Crimson Hexagon, are analyzing social media as a valuable new source of customer insight. As Chris Moody, Twitter's vice president of data strategy, put it, "Twitter is the world's best focus group."
3. It's really about culture more than technology.
You need some people with deep quantitative skills, but what you need most is to foster a mind-set across the organization. Thinking about data should be a first principle, the starting point of inquiry. What story does the data tell you?
The promise of the modern data world is that the story is far richer and more detailed than ever before, making it suddenly possible to see more and learn faster. Donald Walker, a senior vice president of McKesson, calls it "making the invisible visible." At McKesson, a drug-distribution giant (about 240 million pills a day), the technology-enhanced vision has helped it reduce the inventory flowing through its network at any given time by $1 billion.
But this isn't a game just for corporate behemoths, by any means. And it doesn't have to involve Google-style volumes of data.
Take the Denihan Hospitality Group, a 14-hotel chain. Years ago, Denihan realized that its data flow was overwhelming its management-by-Excel spreadsheet system. It has built, with outside help, a modern data infrastructure that combines all its internal data on bookings and rates with outside sources--like web-scraped data on pricing and trends at competing hotels. Its business-trend software, delivered daily to hotel managers' laptops or smartphones, shows the current status of each hotel and serves as an early warning system that looks out three months or more.
Individual hotel managers estimate the resulting improvement in productivity and revenue at double-digit levels. Brooke Barrett, co-chief executive of the specialty hotel chain, describes the change more in terms of the culture of decision-making. "It encourages people to ask all sorts of what-if questions and see things differently," she says. "As a mentality for making better, more grounded decisions it has been critical. And it's been a journey."
4. Be guided by data, not ruled by it.
Big data can be a powerful tool, but remember it has its limits. Two quotes frame the issue succinctly.
"You can't manage what you can't measure." It's most often attributed to W. Edwards Deming, the statistical quality-control guru, or Peter Drucker, the management consultant. But who said it first doesn't matter as much as what it says--a basic truth, for sure.
The second quote is usually attributed to Albert Einstein, but the stronger claim apparently belongs to the sociologist William Bruce Cameron: "Not everything that can be counted counts, and not everything that counts can be counted."
In the age of big data, that second quote is worth keeping in mind as well as the first.