In this video, Kip Boyle looks at one of the steps in information discovery— preparing to collecting scores. Learn what preparations an organization must make to collect scores from information security program experts.
- [Instructor] Let's prepare to collect scores from your experts. We'll start by setting up a single place to record the scores you'll gather. This is your working document, and not meant to be a questionnaire. In the future, I'll explain the workflow for collecting the data. Put your experts' names with the controls they will score into your spreadsheet. Here you can see, I have the recovery controls from the NIST Cybersecurity Framework listed by their unique codes in the first column called ID. Then in the second and third columns to the right, I have listed two experts by name along with their titles.
Notice Bob will not score three of the controls, so I've colored the intersecting cells black, and will not include them in the calculation of the summary scores. Because I'm using the NIST Cybersecurity Framework, the three columns to the right are labeled Outcome, Activity, and Function, which are the three levels of controls defined in that framework. Since you already know who will be providing scores for each control, making your score sheet should take you about two hours unless you have dozens of experts to interview. This is what your data might look like after you collect all the scores.
Notice the colors of the cells follow along with the score key. In Excel, you can use conditional formatting to make that happen automatically. Also notice the scores in the three right-hand columns are calculated as simple averages, also called the arithmetic mean. Let's step through the roll up of the averages. The Outcome Average is the mean of Alice and Bob's scores for each outcome. The Activity Average is the mean of the outcomes for each activity, and the Function Average is the mean of all the activities.
Your next major decision is to choose the level of data quality you need. This will be determined by several factors. Your organizational risk appetite, management style, available budget and timeline, and your internal culture. Make sure to discuss this choice thoroughly with your boss, as it's crucial to have their support when the data is analyzed, and you bring back the top risks. In my experience, there are three levels of data quality to choose from.
With the good quality option, you will briefly train your experts, and they will submit their scores using an online survey tool, such as SurveyMonkey, or maybe you'll use an internal survey tool. When you have dozens or hundreds of experts to work with, this is the most scalable, and least expensive data gathering option. It will generate actionable data assuming your experts participate fully and sincerely, without fear of retaliation for pointing out problems. Some people have doubts about the usefulness of self-reported scores, and there is some cause for concern.
Self-reported answers may be exaggerated, and some experts may be too embarrassed to tell the truth, thinking it will reflect poorly on them and their work. Also experts are inherently biased by their feelings at the time they fill out the questionnaire. You can address these concerns with additional training, arranging for their supervisors to encourage honesty, or by choosing one of the higher data quality levels. With the better quality option, again you briefly train your experts, and then interview them, either in person, or by voice or video call.
This approach requires two data collection hours per expert. This choice is good when you suspect the experts are unable to participate fully and sincerely using the online response method. The interview allows you to watch for signs during your interaction that your expert is giving you biased responses. If you suspect that's happening, take a curious tone and ask the expert to explain more fully why they're giving you that particular score. The more they talk, the closer you'll get to reality.
With the best quality approach, you briefly train your experts, and then interview them, either in person, or by voice or video call. Then, they provide evidence to justify their scores, which you should gather as soon as possible. This approach requires 50% to 100% more data collection hours than the better quality choice, mostly for follow-up to collect the evidence. You might also choose this option as a readiness assessment for an upcoming audit or standards-based certification.
- Goals and components of an information security program
- Measuring and managing information risks
- Reducing risks to an acceptable level
- Using a workflow to organize your work
- Communicating progress with executives and stakeholders
- Demonstrating compliance