In this video, Kip Boyle looks at the workflow structure for information discovery. Learn how to set up an effective workflow to collect scores from information security program experts.
- [Instructor] Let's look at the workflow to collect scores from your experts. You can begin this work after you identify your experts and your questionnaires have been created. When gathering data using an online tool, under the good quality option, your next step is to prepare for questionnaire distribution and tracking. You'll know you're done when your experts can remotely access and complete the questionnaire, and you can analyze the data. Expect a lot of upfront work, setting up the online system, but the payback will happen when you don't have to travel overnight or spend lots of hours conducting interviews.
The more experts you have to collect data from, the greater the return on investment. It can take between 30 and 40 hours of your time, over a period of three or four weeks to convert the questionnaires into your online system, do basic testing, and send the notifications. You may need help from an expert on whatever survey system you're using. Before you distribute the survey, recruit someone to receive a notification and try the questionnaire. However, when you're gathering data by conducting interviews using the better or best quality methods, instead of preparing an online survey, your next step in the workflow is different.
You start by designing the interviews. This will take between two and three hours of your time. You'll need to decide if you want to interview one expert at a time, or you could also meet with several experts from the same work group, and generate consensus scores. Depending on time, distance, and budget, the interviews could be in person, or done by conference call or video call. Be sure to prepare enough score keys and questionnaires so every expert will have their own copies during the interview. I've gone as far as to print the materials out and send them in advance.
Instead of printing handouts, you might want to show the information on a monitor or projector. That works fine in many situations, but could be awkward during the interview unless the experts can see the questions and the score key at the same time, but you may be able to do it if you carefully plan to have the right equipment available in the room for every interview. With either the online or interview-based approaches, your next step is to prepare the expert's supervisor for the meetings. Ideally, this is a meeting or phone call that will take between 15 and 30 minutes per supervisor.
You can save time if you can speak with multiple supervisors all at once. Ask the supervisors to tell their experts ahead of time that the purpose of the scoring is to find opportunities for improvement, not to catch people doing things wrong and get them into trouble. Mention that the raw data gathered will not be shared with executives, only the summary data. Ask the supervisors to encourage their expert to tell the respectful truth when assigning scores. To increase the likelihood the expert will score accurately and truthfully, ask their supervisors to not attend your meeting with their expert.
If they insist on attending the interview, ask what it would take for them to feel comfortable not attending, and then try to meet their request. If they continue to insist on attending, you should let them, or select a different expert. After giving the supervisors a few days to talk with their experts, send a meeting request to each expert to gather scores. Budget 15 minutes per expert to set up appointments. Expect to spend up to two hours per meeting with experts to gather scores. The main drivers of interview time are the number of people, and the number of questions.
Every interview is a balancing act. Keeping an eye on the clock, while trying to not rush through it. If you have concerns about how it will go, do a practice interview with a friend. Here's the standard agenda that assumes a 90-minute meeting. At the start of the meeting, using words similar to those you said to their supervisor, take 10 minutes to briefly explain the purpose of the meeting, then briefly explain how the data will be used after it's collected. Next, pass out the supporting materials and give the experts a 20-minute training.
Show them how the score key works. When determining a score, ask them to consider what they have experienced with that control in the previous six to 12 months, and what they expect will happen six to 12 months into the future. Note, this approach emphasizes learning to score as you go. Experts may feel awkward at first using this approach. Your job during this early phase is to minimize your own talking, and give them time to figure it out. Finally, step through the questions one by one, and collect scores over the final 60 minutes.
Read each question out loud while the expert follows along. After each question is read, allow silence to enter the room, to give the expert a chance to consider the question and use the score key. If the expert asks for clarifications, do your best to help, but let the expert assign the score. Document all scores in your spreadsheet as you go, including any insightful comments made by the expert. Let's look at a few time estimates for gathering scores from experts. These estimates do not include the setup effort.
Using the good data quality method, I spent 46 hours over four weeks collecting data online from over 150 experts across the world at a $1 billion financial services company. On another occasion, using the better data quality method, the chief information security officer at a $4 billion county government spent 29 hours over five weeks interviewing 22 experts. And finally, using the better data quality method, I spent 12 hours over three weeks interviewing experts at a $2 million non-profit agency.
- Goals and components of an information security program
- Measuring and managing information risks
- Reducing risks to an acceptable level
- Using a workflow to organize your work
- Communicating progress with executives and stakeholders
- Demonstrating compliance