Learn about the 5 levels of evaluation and how to measure impact of learning after iterations.
- I now want to get into the nuts and bolts of running your L&D program. Whether you're a team of one or 100 there's some common elements to address. These may flex in terms of size, scope, and budget, but the basic elements and considerations are the same. First, I want you to consider how you'll evaluate the impact of your learning. Did you know that there's actually five levels of evaluation? The success of your program depends on your ability to demonstrate all five. These were initially identified by Donald Kirkpatrick, but were updated by Jack Phillips who runs the ROI Institute.
As we go through them let's apply them to our example with John and Maria at the hospitality company. John has contact Maria because his marketing team isn't delivering the right level of quality on time. Maria runs L&D and built a learning solution to address his needs. Let's look at the first, most basic level of evaluation, which is how satisfied the participants were. Did they like it or not? After the training Maria would ask participants how much they liked or were satisfied with the program.
Some people use net promoter score, or NPS, which is an example of this lowest level of evaluation. I also like to ask learners to comment on how valuable the learning was to them, and whether they improved their knowledge and skill. The second level measures comprehension, and whether people learned the content. This can only be truly measured through an assessment, since self-reports can be inaccurate. But it's very easy to use survey tools like SurveyMonkey or interactive videos like HapYak to assess understanding.
Maria could assess comprehension on executing the project plan, time management, and quality control. Even better, she could roll out post-event booster material if the results showed low comprehension on an element. The third level is implementation, and measures if people were able to deploy that learning on the job. I'm sad to say that this is where a lot of L&D programs fall down, which is why thinking like a consultant is so important. It allows you and your stakeholders to get clear about what behaviors you need to see on the job, and by when.
You can track this by observing their behaviors or by looking for changes in the metrics you identified. John should be able to observe and track change in his team's behaviors. The fourth level is about impact, and whether that behavior change moved the needle on the metrics that matter. In our example we should see the percentage of on-time delivery of marketing materials increase at the same time we see percentage of errors decrease. If that's not happening, Maria would work with John to adjust the learning solution or add more elements until the results are achieved.
The bottom line is if your organization is investing time and money on learning, then it needs to demonstrate the necessary results, period. Finally the last level is calculate ROI or return on investment. It's a measure of the cost spent on the learning solution as it relates to the savings created by the behavior change. There are three different things you can measure here, and they can be calculated with any currency. First, you can measure the benefit-cost ratio, which is the program benefits divided by the costs.
Let's say Maria's learning solution cost $20,000 to roll out, but over time it saved $35,000. Then the cost-benefit ratio would be 1.75:1, meaning for every dollar she spent the organization received $1.75 in benefit. That's a good return. The second is ROI, which is expressed as a percentage. You take the program benefits minus the program costs, and then divide that by the program costs.
Using the same example the ROI was 75%, meaning that every dollar gave back 75% more in benefit. Finally, there's the time to payoff, which is a measure of how long it took for the investment in learning to pay for itself. You take the program costs and divide them by the program benefits, and then multiply that by a unit of time. In this case we'll use 12 months of the year, so this learning solution paid for itself in just over six months. After that it's all benefit.
Now, in these examples we were tracking actual financial cost of money spent and saved. But all kinds of costs and benefits can be explored. You can look at cost, output, time, quality, and energy, all of which can be translated to financial data. And there are other people indicators that have a crucial impact on your business too, including customer service, creativity or innovation, the development of your talent, and the culture of your organization.
So, be thorough when you explore the metrics that matter in how you can show the impact of your learning solutions. As learning professionals we must design learning to deliver on all five levels of evaluation and track the results. This not only helps us show real ROI on learning, but we can course-correct as needed, so that we're always delivering the metrics that matter to our organizations.
- Identify the six stages of organizational development.
- Describe how to recognize your organization’s L&D stage.
- Explain how to create a culture of learning in an organization.
- Summarize important aspects of adult learning theory.
- Recall the levels of Bloom’s Taxonomy.
- Recognize the importance of assessing your audience prior to training.