LIMITATIONS AND FUTURE WORK

The exploratory nature of the open-ended interviews and the small sample size are limitations of this study. In order to develop a richer understanding of the usability and value of the information visualization system to participants of cyber defense competitions, more field interviews need to be gathered. In addition to interviews, formal usability testing of the visualization system should also be conducted. Many participants noted that their performance in the competition improved by utilizing CDCVis, however, those individual reports should be corroborated by quantitative measures.

Various changes to the system have been instituted since this work has been completed. First, a ‗how-to‘ document was written and has been provided to users at the start of subsequent competitions to allow for better understanding of the system. Similarly, the scoring metrics of the competition have been changed in response to user feedback so that higher scores are now better and demerits are applied negatively. A written explanation of the scoring has also been provided to participants in subsequent competitions.

REFERENCES

Aiken, L. (2002). Attitudes and Related Psychosocial Constructs: Theories, Assessment, and Research. Thousand Oaks, CA: Sage Publications.

Beyer, H., & Holtzblatt, K. (1998). Contextual Design: Defining Customer-Centered Systems. San Francisco, CA: Morgan Kaufmann Publishers, Inc.

Chamalese, G., & Pridgen, A. (2004). The Success of the UT IEEE Communications Society. Paper presented at the 8th Colloquium for Information Systems Security Education. Conklin, A. (2006). Cyber Defense Competitions and Information Security Education: An

Active Learning Solution for a Capstone Course. Paper presented at the Proceedings of the 39th Annual Hawaii International Conference on System Sciences, 2006. HICSS '06.

Cowan, C., Arnold, S., Beattie, S., Wright, C., & Viega, J. (2003). Defcon Capture the Flag: Defending Vulnerable Code from Intense Attack. Paper presented at the 2003 DARPA Information Survivability Conference and Exposition.

Dodge, R. C., & Ragsdale, D. J. (2004). Organized Cyber Defense Competitions. Paper presented at the Proceedings of the IEEE International Conference on Advanced Learning Technologies.

Few, S. (2006). Information Dashboard Design: The Effective Visual Communication of Data. Sebastopol, CA: O'Reilly Media, Inc.

Foresti, S., Agutter, J., Livnat, Y., Moon, S., & Erbacher, R. (2006). Visual Correlation of Network Alerts. IEEE Computer Graphics and Applications, 26(2), 48-59.

Goodall, J. R. (2005). User Requirements and Design of a Visualization for Intrusion Detection Analysis. Paper presented at the Proceedings of the 2005 IEEE Workshop on INformation Assurance and Security, United States Military Academy, West Point, NY.

Goodall, J. R., Lutters, W. G., & Komlodi, A. (2004). The Work of Intrusion Detection: Rethinking the Role of Security Analysts. Paper presented at the Proceedings of the Tenth Americas Conference on Information Systems, New York, NY.

Goodall, J. R., Lutters, W. G., Rheingans, P., & Komlodi, A. (2005, Oct. 26, 2005). Preserving the big picture: visual network traffic analysis with TNV. Paper presented at the IEEE Workshop on Visualization for Computer Security, 2005. (VizSEC 05), Minneapolis, MN.

Hoffman, L. J., Rosenberg, T., Dodge, R., & Ragsdale, D. (2005). Exploring a national cybersecurity exercise for universities. IEEE Security & Privacy, 3(5), 27-33.

Jacobson, D., & Evans, N. (2006). Cyber Defense Competition. Paper presented at the 2006 ASEE Annual Conference & Exposition: Excellence in Education.

Komlodi, A., Goodall, J. R., & Lutters, W. G. (2004, 2004). An Information Visualization Framework for Intrusion Detection. Paper presented at the Conference on Human Factors in Computing Systems, Vienna, Austria.

Komlodi, A., Rheingans, P., Ayachit, U., Goodall, J. R., & Joshi, A. (2005, Oct. 26, 2005). A user-centered look at glyph-based security visualization. Paper presented at the IEEE Workshop on Visualization for Computer Security, 2005. (VizSEC 05), Minneapolis, MN.

Kuniavsky, M. (2003). Observing the User Experience: A Practitioner‘s Guide to User Research. San Francisco, CA: Morgan Kaufmann Publishers, Inc.

Lakkaraju, K., Bearavolu, R., & Yurcik, W. (2003, 2003). Nvisionip – a traffic visualization tool for security analysis of large and complex networks. Paper presented at the International Multiconference on Measurement, Modelling, and Evaluation of Computer-Communications Systems (Performance TOOLS).

Lakkaraju, K., Yurcik, W., Bearavolu, R., & Lee, A. J. (2004, 10-13 Oct. 2004). NVisionIP: an interactive network flow visualization tool for security. Paper presented at the 2004 IEEE International Conference on Systems, Man, and Cybernetics, Urbana, IL.

Lakkaraju, K., Yurcik, W., & Lee, A. J. (2004, 2004). NVisionIP: netflow visualizations of system state for security situational awareness. Paper presented at the Proceedings of the 2004 ACM workshop on Visualization and data mining for computer security, Washington DC.

Livnat, Y., Agutter, J., Moon, S., Erbacher, R. F., & Foresti, S. (2005, 17-19 June 2005). A Visualization Paradigm for Network Intrusion Detection. Paper presented at the Proceedings of the 2005 IEEE Workshop on Information Assurance and Security, United States Military Academy, West Point, NY.

Luse, A., Scheibe, K. P., & Townsend, A. M. (2008). A Component-Based Framework for Visualization of Intrusion Detection Events. Information Security Journal, 17(2), 95- 107.

Oline, A., & Reiners, D. (2005, Oct. 26, 2005). Exploring three-dimensional visualization for intrusion detection. Paper presented at the IEEE Workshop on Visualization for Computer Security, 2005. (VizSEC 05), Minneapolis, MN.

Richardson, R. (2008). 2008 CSI Computer Crime & Security Survey. Computer Security Institute, 1-30.

Riding, R., & Rayner, S. (1998). Cognitive styles and learning strategies. London: David Fulton Publishers.

Schepens, W., Ragsdale, D., & Surdu, J. R. (2002). The Cyber Defense Exercise: An Evaluation of the Effectiveness of Information Assurance Education. The Journal of Information Security, 1(2).

Schepens, W. J., & James, J. R. (2003). Architecture of a Cyber Defense Compeition. Paper presented at the IEEE International Conference on Systems, Man and Cybernetics. Shneiderman, B., & Plaisant, C. (2005). Designing the User Interface (4th ed.): Pearson

Education, Inc.

Vigna, G. (2003a). Teaching Hands-On Network Security: Testbeds and Live Exercises. Journal of Information Warfare, 3(2), 8-24.

Vigna, G. (2003b). Teaching Network Security Through Live Exercises. Paper presented at the 3rd Ann. World Conf. Information Security Education (WISE 3).

Yin, X., Yurcik, W., Treaster, M., Li, Y., & Lakkaraju, K. (2004, 2004). VisFlowConnect: netflow visualizations of link relationships for security situational awareness. Paper presented at the Proceedings of the 2004 ACM workshop on Visualization and data mining for computer security, Washington DC.