Program evaluation

- [Instructor] Why are some entrepreneurs more successful in business than others? Are they smarter, more driven, or do they have better opportunities? You're likely to spark a debate if you try to simplify Steve Jobs' success by saying he was just smart. If we want to mimic and recreate success, we must first understand why it happened. For learning analytics, that means finding the source of success. Let's take a real-world scenario in which we work for Sauder Manufacturing and we're training managers to take advantage of newly implemented enterprise planning software. We've successfully executed the program and measured improvements in financial and operational metrics, but now the challenge is that we have to plan next year's program with less budget. But we need to evaluate the learning program to ensure we don't cut something hugely successful. Learning program evaluation is the act of determining why your learning program was successful. Designing a measurement plan based around a program logic model makes it possible to determine whether your program was successful, but we have to dig a bit deeper to determine why it was successful. To answer a question of why a learning program worked or didn't work, it's best to start with identifying relationships between inputs and resources. This will be learning experiences available to a learner population and the results and outcomes of the learning program. For example, as part of evaluating our managerial operations program at Sauder Manufacturing, we need to know the relationship statistically speaking between data about courses, coaching, and collaboration and data about changes to their ability to manage inventory, use new software, and build teams. At the end of the program, we can isolate the managers who have shown the greatest skill growth. What do the people who benefited the most do differently from the other program participants? For instance, maybe they engaged with a greater percentage of collaborative learning opportunities in addition to the mandatory, formal courses. If we're able to draw a correlation between those learners who experienced high growth in their skills and those who also engaged in a larger portion of collaborative discussions, we made deduce that our learning program was successful because of the additional informal learning that was provided to augment formal courses. Even though correlations don't ensure causality, they help us understand relationships between two variables. With environmental knowledge of what's happening at your organization and a well-defined logic model, the correlations in the previous example go a long way to answering why a particular learning program was successful. If we isolate which learners had the greatest degree of success or succeeded the fastest, we can use a regression analysis against the inputs and resources available to those people to see why they were so successful. What was the last learning program a group of people at your organization completed? Whether it was a resounding success or a catastrophe, find the rock stars or flame outs. Using statistical methods like regression analysis or just interviews with the learners, learn what was uniquely impactful. Doing so will prepare you with knowledge of what works best for future implementation.