7.1 Appendix 1: Interview with SOC experts
About the person
1. Please give us a little background on you and your job. (education, since when do you work at your current place, why)
2. Please walk me through your typical working day.
3. What kind of tasks are you responsible for? (tasks, time, size, interfaces) (Daily scan/ Compliance check/ Troubleshooting)
4. What is the process that you are following? (How it escalades from SOC Level 1 to SOC Level 3) 5. What is the area of responsibility for monitoring? (Global / Local)
6. How do you keep yourself “up-to-date” with the new vulnerabilities? (Eg. Start the day by examining a collection of blogs & websites to find new vulnerabilities
7. Are there any particular elements which you have to keep an eye on during the day?
About the environment
8. What kind of alerting tools or websites do you use? 9. How often do you check them?
10. How critical is the information on those? 11. What is your next step after that?
12. For what kinds of tasks do you use dashboards?
13. What would be the most important things that you would like to see on a dashboard? 14. In what context (when you open your computer / on the / the state of the system)?
About the job
15. What is the best thing about your job? What do you like best? 16. What do you like least with your job? What is the most stressful part? 17. What skills are required to do your job?
18. How did you get into security? (school/ self-taught/ games) 19. What do you like about this field the most?
About the tools
20. What tools/applications are you using to complete your tasks? 21. What do you like, and what do you dislike about those tools? 22. Approximately how much time does it take to complete such a task?
23. What does completion mean in this case? What do you do with the information once you got it?
7.2 Appendix 2: Usability Test Dashboard
Hey. My name is Andra Cimpan and I am going to be walking you through this session today. My colleague Virag will help us by taking the notes.
Before we begin, I have some information for you, and I’m going to read it to make sure that I cover everything.
INTRO
You probably already know why we asked you to come here but let me go over it again briefly. During the last months we have been working on creating a new concept of dashboard that would suit the needs of our users and we would like to do a first usability test in order to validate the prototypes. DURATION
The session will take about 60 minutes and I’ll break it into 3 parts. In the first part we will take a look at the application where I’ll ask you to perform some given tasks on the prototype. In the second part I will ask you some general questions regarding the system that you experienced followed by the last part, in which I will ask you to fill in a survey.
TESTING THE APP, NOT YOU
The first thing I want to make clear right away is that we’re testing the applications, not you. You can’t do anything wrong here. Also, the current system is just a prototype and not all the functionalities have been implemented.
THINK OUT LOUD
As you use prototypes, I’m going to ask you as much as possible to try to think out loud: to say what you’re looking at, what you’re trying to do, and what you’re thinking. This will be a big help to us. Also, please don’t worry that you’re going to hurt our feelings. We’re doing this to improve the applications, so we need to hear your honest reactions.
QUESTIONS
If you have any questions as we go along, just ask them. I may not be able to answer them right away, since we’re interested in how people do when they don’t have someone sitting next to them to help. But if you still have any questions when we’re done, I’ll try to answer them then. And if you need to take a break at any point, just let me know.
PERMISSION TO RECORD
With your permission, I’m going to record what happens on the screen and our conversation. The recording will only be used to help us figure out how to improve the app, and it won’t be seen by anyone except the people working on this project. And it helps me, because I don’t have to take as many notes.
QUESTIONS?
Do you have any questions so far? OK. Before we look at the site, I’d like to ask you just a few quick questions.
TASKS
1. You are coming to work Monday morning and get straight into work. Can you describe the state/status of the system after the weekend? 2. What would be the first thing that you would check?
3. Do you see any major changes that happened during the time that you were away? Which are the days with the peak number of incoming events? What are those numbers?
4. Identify the lowest compliance score of a policy set and get more details about that. 5. Create a summary of the activity of the team from the last week. What would you include?
QUESTIONS
1. Which one of the two presented dashboards do you think you would use more for your work? 2. What are the main problems and difficulties you have found while using this prototype? 3. What or which functions have you liked most of the prototype? Why?
4. Can you describe your overall experience with this prototype? 5. How did you like the user interface design?
6. How would you evaluate the set of functions being offered in the prototype? 7. What would you improve in this prototype?