DATA COLLECTION

Subjects for the usability portion of this research consisted of teams of individuals who participated in a CDC at a large Midwestern university. The teams consisted of college students pursuing a major involved with network and security administration. Teams came from eight different colleges/universities across the state. School size varied from community colleges to a Research University with very high research activity.5 Teams were comprised of four to seven members. This group was solicited due to the live nature of the field study and the range of different participants. Participants were provided with a scenario one month before the competition which detailed a fictitious corporation they must setup and

5 _{This is according to the Carnegie Classification of colleges and universities which can be found at}

administer. Each team was allowed to use four machines with any legally obtained software they deemed fit, which was provided by the host university. The teams were given one month prior to the competition to remotely setup their networks utilizing a network-based KVM, and were also allowed to come in one day prior for any final setup needed. The competition lasted from Friday at 5:00 p.m. until Saturday at 11:00 a.m. during the spring semester.

The study of CDCVis was undertaken about seven hours into the competition around 12 midnight so the participants had become accustomed to the competition but were not yet drained from the all-night contest. One of the developers of CDCVis was on-hand and answered any questions about the system both before and after the study. Two interviewers questioned the students utilizing only a PDA recorder and a pad of paper.

During the 18-hour Collegiate Cyber Defense Competition, six teams were interviewed. Seventeen individuals contributed to the discussion of the newly introduced visualization system. The purpose of the unstructured interview was to

1. explore the general attitudes of the group about the visualization system 2. assess how the visualization system was used during the competition

3. discover what problems existed which might suggest further development iterations

Attitudes were defined as the tendency to respond positively or negatively to a given person, situation, or object (Aiken, 2002). Usage was simply measured by how the system helped participants accomplish the task at hand. Problems were measured by the expressions and reports of frustration.

Nearly two hours of discussion was recorded which resulted in 145 statements from the participants about the visualization system. The interview data was coded following recommendations offered by usability researchers (Beyer & Holtzblatt, 1998; Kuniavsky, 2003). An affinity diagram approach (Beyer & Holtzblatt, 1998) was used to reveal common issues and themes.

RESULTS

Five of the six teams had prior experience participating in previous Collegiate Cyber Defense Competitions. However, none of the teams had used a visualization system to support their defense activities. Common themes emerged suggesting how the visualization system was being used and what problems were experienced.

Each of the six teams said that they used the visualization system to check on their scores. Three teams said that they used the system to check on their services. Another team stated that they used the visualization system to help them improve their response time, to discover their service vulnerabilities, and to focus on the task at hand.

 Now I look up there whenever to see your scores. [Team2]  It helps seeing all the services. [Team 5]

 It definitely helps with the response time. What exactly we were vulnerable to. What we really needed to be looking at. And what‘s not important to be looking at. And it helps us cut down on what we don‘t need to worry about either. [Team 3]

Three teams expressed problems with interpreting the scoring. They were confused by the graph of negative and positive numbers. The problems were resolved by asking other teams for clarification. The confused teams were then assured that the negative numbers were good scores and the positive numbers were demerits.

 I went over and looked and then said, ‗are high numbers good or bad?‘ [Team1]

 But once I figured out that the negatives were better, then it was pretty simple. [Team2]

 At the beginning we asked around a little bit because some of the things weren‘t clear. [Team 6]

Two teams expressed problems with the program‘s interface.

 Yeah, when you try to click on ‗status‘ there‘s no way to go back that I‘ve found. When you try to click the ‗back‘ button you have to log back in. That‘s really annoying. [Team 4]

 It‘s just a pain in the ass when I‘m sitting here going, ‗refresh!‘ [Team 4]

Two teams requested a user guide to assist them with interpreting the visualization graphics.  Some kind of user guide, I guess, would have been nice. [Team1]

 Are there any documents saying what each zone is about and how to use them? [Team 3]

Despite the problems experienced, the teams expressed more positive attitudes toward the visualization system than negative. Of the 79 statements that directly referred to the visualization system, 55 of those statements were positive and 24 were negative.

 Would it be possible to get this kind of thing running at our school? We would definitely be interested. [Team 1]

 It‘s all pretty cool. [Team 2]

 The visualization helped us focus more on what exactly the problem was at the time. [Team 3]

 It‘s definitely not something I would want to get rid of. There are just tweaks and of course, that just comes with time. [Team 4]

 I think it would be real cool if we could use it at name of school. [Team 5]  This is my second year coming to competitions and I feel like it‘s really neat

to just see, ‗oh shoot‘ we‘re going to get hit with this sort of traffic or whatever. [Team 6]

DISCUSSION

The investigative study of the usability of the newly introduced visualization program provided valuable information as to how the participants appropriated this technology into their overall team strategy. Training materials had not been decimated before the competition pertaining specifically to the visualization system. The feedback received from the user interviews lead to the development of a ‗how-to‘ CDCVis document for future competitions. Even though the field interviews and user observation revealed that the

visualization system could be improved by further iterations, the user feedback was more positive (71 percent) than negative.

There were challenges noted with the display of information. The scoring schema reproduced in the graphs was not intuitive to the users. Teams were initially confused because the graph displayed positive team scores with negative numbers and team demerits with positive numbers. However, once participants understood the scoring schema they were no longer confused and were able to interpret the display. There were also suggestions offered from the interviewees about the general usability of the program‘s interface. Users requested quicker ―refresh‖ times and a more visible ―back‖ button to allow them to return to the main views.

These new users learned to interpret the team and network views in the first hour of exposure to the information visualization program. The teams then used the visualization to check their scores and to discover higher-level threats and vulnerabilities. The team and network views were used to assist competition participants with decision making and performance. Several teams noted that their performance improved because their response times decreased. In addition to improving team performance, several interviewees also noted that the visualization system reduced the stress of the competition because the display allowed them to focus on the immediate problems and ignore the periphery, non-threatening activities.

The user interviews concluded by half of the teams inquiring if the visualization system would be used in future cyber defense competitions. The qualitative interviews and field observations indicated that the visualization system added value to the competition.

New users were able to quickly interpret the team and network views and were able to appropriate that information in their overall strategy.

CONCLUSION

This purpose of this study was to look at the utilization of visualization systems for computer and network security. Specifically, this research explored the types of visualization systems used during a cyber defense competition. The study has two main contributions. First, it details a visualization system which has been implemented for a current cyber defense competition. This information can be used by others who are also interested in developing such a system for use during similar competitions. Second, the research provides a first-pass look at the usefulness of the system by the users. This provides initial insights into the educational impacts of the system and how the system can be better leveraged to accomplish these objectives.

While not direct, this study in this type of environment provides a corollary to actual corporate network security administrators utilizing visualization systems for computer and network security. As with many corporate situations, it is difficult to adequately conduct research in actual production environments. CDCs provide a valid testing ground for field experiments in the area of corporate network security. This study capitalizes on this environment to provide an initial look at the use of visualization systems in corporate environments for network security.