What has happened? What is happening? What might happen next? What's happening across my company and supply chain? This video covers using systems that look backward, forward, and across.
- Before you roll out any system of metrics you need to test the system. This requires your experience and your imagination. You need to imagine common scenarios, unlikely, but realistic scenarios, and you need to consider how someone might try and game the system. Consider this grading scale for students in a college course. Exams are each worth 20% of the grade, homework and attendance are each 10% of the grade, the team project is 30% of the grade, and on the team project the team members will evaluate your performance, that's 10% of the grade.
This looks reasonable. The instructor wants the student to master the material, work outside of class on homework, attend the lectures, and be a good teammate on the semester project. But will the outcomes this system generates by fair? Will they actually produce grades that the instructor feels are just? In order to properly test the system we first need to consider some of the possible scenarios. Let's quickly look through four possible scenarios.
Scenario one, a student does well on individual work, but the student gets placed on a very poor team. While this student produces excellent work for the team the team project is poor, because of the lack of effort on the part of the other three members. Does this student really deserve a C in the course? Perhaps the team project and the rules of the peer evaluation need to be investigated before going ahead with this grading system.
Scenario two, a student does well on the exams, 90s, but they fail to attend lecture or be a valuable member on the team assignment. They get 0s on the homework and attendance, and only a 50 on their peer evaluation. This student gets just a bit above a 70% in the course. They'll get a C in the course. The student studied for exams and was placed on an excellent team that did well in spite of his relatively poor team effort.
Again, we may need to consider the importance of the weights on homework, attendance, and the peer evaluation. Let's try another scenario. Scenario three, a mediocre student that decides not to participate on their team assignment, 70s on most of their assignments, a 95 on a team assignment, a team assignment that they did not do any work on. Notice, they got a 0 on their peer evaluation.
So the student was just barely okay on exams, homework, and even in attendance, then they got placed on a great team, but they did nothing to help the team. Does this student deserve a C in the course? One final scenario. A student tries very hard, they attend all the lectures, the students peers say the student did excellent work on the team assignment, but the student performs horribly on tests.
Should effort and team contribution be enough to help a student that doesn't seem to grasp the material get a C in the course? What do we learn through our little tests? While the weights on each course component seem fair there are some ways in which good students can get cheated and other cases where bad students can pass the course. Plus, we learn a little bit about how certain individual and team assignments should be structured.
Testing your system of metrics can help uncover some problems with the system. The test may also provide guidance on assessment and measurement. And finally, the test may also help us better understand how to account for possible victims, as well as cheaters that may look to take advantage of the system.
- Metrics and human behavior
- Common corporate errors in measuring
- Developing a good metric
- Using the performance measurement tune-up
- Avoiding redundancy
- Using dashboards, infographics, and other data visualization tools