Note: Because this is an ongoing series, viewers will not receive a certificate of completion.
Skill Level Beginner
- [Instructor] How many times have you run a usability test after your product has shipped? It may seem like a crazy idea. After all, it's too late to change anything. But there's some real value to running a benchmark usability test. Benchmark testing gives you regular checkpoints to ensure you're improving the usability of your product with every release. Although benchmark testing follows a very similar process to regular usability studies, it has a different purpose. Rather than diving into a specific part of the product, such as a set of new features you've added, benchmark testing runs the same general tasks each time. This lets you track your product's performance with each set of new features you add. Rather than collecting data on where the showstopper issues lie, benchmark testing gathers a common set of metrics on efficiency, effectiveness, and satisfaction. This way, it's possible to compare the usability of the product over time and track how well each release is performing. For example, say you're working on a team that's building a flight-booking app. During development, you might run usability tests on new features you're adding, such as the addition of a price comparison feature or changes to the payment process. When you run benchmark studies though, you'll have a set of higher-level tasks aimed at testing whether the product is meeting the product team's overall goals. These goals don't change from release to release. For instance, it's always important that people can research destinations and buy tickets with your app, regardless of the specific features that were added in the last development cycle. The team may also have goals around making the experience exciting, entertaining, trustworthy, or fast to complete. Running a benchmark study after each major update to the product allows you to track these goals across time. Typically, you give participants the same tasks each time you run a benchmark study and compare the results between product versions. Like I said, the results typically fall into three categories. Efficiency is how easily people can complete the tasks. Normally, that's measured by time on task versus time recovering from errors. Effectiveness is how well they manage to complete their tasks. Did they book the correct flight? Did they book it for the right number of people? Satisfaction is obviously how good they feel about the outcome. As I mentioned in my full course on usability testing, the typical formative usability studies you run during development will have around eight participants. With summative benchmark studies, you need to be sure that you're capturing valid data, which typically means running more participants, sometimes 20 or more. It can be boring to run that many participants, especially when you aren't necessarily learning about any new usability issues. It can also be harder to justify the expense when the team doesn't see immediate results from the sessions. One solution is to use a local vendor or one of the online testing companies to run each benchmark study for you. Because you don't need to be watching for new insights during each second of the sessions, it's easy to farm this work out to a third party. And remember, once you have a set of common benchmark tasks, there's no reason why you can't also run these task against a competitor's product. That will show you how well they measure up against your experience. I've created a separate video on competitive testing if you want to learn more about that. Benchmark testing can be really worthwhile because it gives you a clear indication of your progress between software releases. That's something which is normally very hard to measure, so benchmark usability metrics can be a good way of making your usability team indispensable to management while also fostering a healthy sense of competition on the team.