In typical sales organizations, a small number of elite reps will sell twice as much as their peers. In most back offices, the best people are getting twice as much work done as their peers. When you ask managers what makes the difference, they often shrug and say, "that's the way it is." But Big Data might just be about to change all that.

Big Data has been getting lots of press as a tool for companies to better understand their businesses and their markets. For sales and marketing organizations, data analysis has delivered better targeted lists, more efficient lead generation, "predictive" pipeline scoring, and even adaptive pricing models. Retailer Target even found a way to use Big Data to predict when customers became pregnant. But while the concept has been a marketer's dream, it has yet to be applied to the area where it could have the most critical impact to companies: employee effectiveness.

Payroll is often the single largest expense for companies, and what they get for that money matters. Yet quantitative employee performance evaluations consistently show wide variations between top performers, average employees, and the laggards. Many employers consider these performance gaps business as usual.

While there will always be differences between the top and bottom performers, the gap is simply too large for employers to ignore. The good news is that Big Data may be able to begin shedding light on performance. More importantly, it may be able to actually help people improve their performance and close the performance gap.

Traditionally, analytics haven't been used to look at individual performance. One reason for this is that individual employees may not generate enough data to analyze. Another is that employee productivity data is typically messy, unstructured, and scattered across many systems. It can be expensive to get and organize. Finally, analysis traditionally requires data scientists, who are too expensive to have spending a lot of time figuring out why Joe isn't selling. Managers and employees, meanwhile, are too busy with their day jobs to invest a lot of time in data analysis.

But technologies have been evolving rapidly. These traditional roadblocks are coming down. Take, for example, the need to get lots of data. A typical office-based salesperson may send and receive more than 100 emails a day, spend hours on the phone and in meetings, and spend hours working in the customer relationship management (CRM) system. All of these systems generate machine data: automatic detailed records of the activities taking place. Although the data in the CRM alone isn't enough for any deep investigation, when combined with all the other information the person is creating, it creates a complete view of how the person is doing their job.

For most types of employees, the machine data they create as they work is enough to begin to answer key questions about why some people are more productive, produce higher quality work, or commit fewer errors. The challenge then shifts from gathering the data to having a data scientist review it.

Fortunately, technology has evolved to the point where managers and employees themselves can explore the data to draw important conclusions about how they can become more effective. One of the hottest areas in this regard is data visualization. The democratization of business intelligence has enabled more and more managers to use graphical data to understand what's happening in their organization. Companies like Domo, Tableau, Good Data, and my business, Enkata, have developed solutions that provide insight without the need for data scientists.

The third limitation is that people are busy. To respond to this, software companies are doing more to put intelligence into the product. Scoring algorithms can evaluate information automatically, and prioritize the things people need to know about first. Pattern-recognition software can automatically find differences in activities that relate to differences in outcomes, and highlight these as things people can change to improve performance.

Diving Into the Data

Can data-driven performance management really change how people work? At Enkata, we're seeing this happen in a variety of ways. For example, people who check their email too frequently don't realize the damage it does to their concentration. People who set their browser's starting page to an entertainment site may not recognize how much time they lose every time they need open a browser while working.

Beyond simple work habit changes, there are actually skills differences that can be found in the data, and these can also be coached. For salespeople, we've seen enterprise reps who over time lose relationships with key decision-makers on complex accounts as people drop out of email conversations. We've seen transactional sales people who disqualify opportunities too quickly and SMB reps who invest too much time in non-responsive accounts instead of moving on. In other work environments, we've seen situations where people don't know the best process, and so struggle where their peers work quickly. With the right tools, these skill gaps are easily identified. Employees who want to improve their performance change quickly once they know what to do.

Management is an art as well as a science. Performance management will still require conversations between managers and their employees. However, as technology matures, these reviews will evolve from drive-by evaluations and periodic check-ups to data-driven conversations about what's really happening, and what specific changes will make people more effective. When this happens, the performance gap will narrow as more people improve their skills and perform at their potential.