Collecting the Right Data

Not all data is created equal -- here's how to know what kind to gather and how to use to make the most of the data process.

We live in a world where data is literally everywhere. And if it is not readily available, we can collect it easily. The challenge is collecting the right data for the purpose. For example, when you implement a program, project, or policy, success is always a goal. So, what type of data would you collect to determine if it was successful? The answer is not always readily apparent. It is best to think of data not only in different categories but actually at different levels. The concept of data levels is helpful to appreciate the relative value of the data sets. For example, if a new software package to help reduce shipping errors is implemented you want to measure success. Obviously, the first thing to check would be the errors. Are they lower? If they have not been reduced then we are left wondering why the software did not work, or if the shipping department did not use it? To address this situation appropriately, let's examine four types of data that should be considered.

The first is reaction (Level 1). As the name implies, this data set is collected by asking the individuals who are involved with the project about their reaction to it. This is not necessarily asking if they like it, but did they find it to be necessary, useful, relevant to what they are doing, or important to their own success. Employees need to see a new project as something of value to either them or the company or, ideally, both. Deciding what reaction we want dictates the type of reaction data that we capture. The process is simple. As the project is implemented, we take a brief survey. Simple questions are asked, such as, "Is it useful? Is it helpful? Can we use it? What are the problems?" and "What will keep us from using it?"

The next level is learning (Level 2). Did the participants learn how to use the software? This is important. Sometimes the new technology is not used because people do not understand it. Obviously, there are instruction manuals and there may be a brief information meeting, and perhaps there is an expert available to help. But sometimes employees are either afraid to ask, do not read well, or do not listen. So we must take a deliberate step to measure learning. Maybe we ask questions, offer a simple quiz, or require a simple demonstration to see that they know the key issues. This helps to confirm that we know the participants know how to use it.

Now, we move to Level 3, (Application). The question at this level is also simple. Did they use it? Application is critical. This involves an employee's use of the information, tools, or content in the project or program. In the case of software, a user profile is needed. Fortunately, many software packages have this built in, but not all. We may have to ask questions, or perhaps monitor work performance. This step is necessary to capture data to understand the extent to which they are using the software properly and the success they are having with it. At the same time, it is important to capture what is getting in the way, what keeps them from using it properly, and what is helping them use it? Identifying these barriers and enablers is necessary to improve the project and to make it more successful. The barriers are always there and may include lack of support, lack of time, or technical problems. The enablers help to make it successful. For example, a job aid, a user-friendly design, or support from the immediate manager are all effective enablers.

Level 4 (Impact) is perhaps the most important data. This data set is the consequence of the application. In our example, it is the shipping errors -- our goal from the beginning. If we collected data on all three previous levels, we have a sense of what should occur. We should see the error rates reduce because employees are now using software that is needed and designed for this purpose. We have a complete profile of success. We have the desired reaction to the software, employees have learned to use it, they are actually using it, it is successful on the job, and consequently, there is a reduction in errors.

So when you want to improve a business measure with a new project or program, think in terms of these four data sets. Visualize a chain of impact as the project moves through reaction, learning, application, and impact. Do not just go to the business data; capture the other data categories so that you can see how the project is unfolding. Data collection at the lower levels often helps to pinpoint problems along the way so that we can make adjustments and corrections.

Incidentally, when we measure the impact (reductions in errors), we must take an extra step to isolate the effects of the project on the data. This was covered in a previous column, Taking Credit When Credit is Due. This is an important step to identify the amount of data that connects our project or program, using a credible method.