In this video, Kip Boyle shows the basics for creating a score key. He discusses how experts measure the strength of each control using the 0 to 10 score key.
- In order to collect reliable data from experts, you'll need a consistent approach that guides them to translate their observations into a numeric score. We'll base our approach on the zero through 10 scoring system we previously learned about, and we'll create a score key for the experts to use. We'll also create a questionnaire for each expert in advance of the data collection. Here's how you create a score key. For each numeric score, zero through 10, prepare a statement or two that describes how well any given control is actually performed.
The number we get from each expert is called the actual score, and we record each one in our spreadsheet. Actual scores are determined by an expert based on how well the organization has performed the control over the past and anticipated future six to 12 months. Here's what my score key looks like at the four major scoring points. Remember that I'm using NIST Cybersecurity Framework, so my lowest level controls are called outcomes. If the expert agrees with this statement, the outcome rarely or never happens when needed, then the score is zero.
What if the expert agrees with this statement? The outcome happens consistently with some minor flaws from time to time. Re-work sometimes happens but is uncommon. Then the score is five. An eight means the expert believes the outcome happens consistently with great effectiveness and high quality. And a 10 means the outcome happens at excessive financial cost to no benefit. Of course, experts are welcome to choose a score between these numbers. Note there are two other possible responses, unknown and not applicable.
Although based on how carefully we prepared for the interview, these responses should be rare. Once your score key is ready, prepare one questionnaire for each expert. Start by turning each control into a question. Start the question with How Well. Let me show you how this works. Here's an original control from the NIST Cybersecurity Framework. The development and testing environments are separate from the production environment. Now, here's the control written as a question. How well are the development and testing environments separated from the production environment? Here's what a questionnaire for an expert would look like.
I've put in one control from each of the five main functions of the NIST Cybersecurity Framework. Notice I've color-coded the functions without using red, yellow, or green. Since those colors have implied meaning, it would likely cause confusion unless those colors were associated with a score. Would you show this version to an expert in a face-to-face interview, or would you dress it up a bit? That's entirely up to you.
- Goals and components of an information security program
- Measuring and managing information risks
- Reducing risks to an acceptable level
- Using a workflow to organize your work
- Communicating progress with executives and stakeholders
- Demonstrating compliance