Assessing the Effectiveness of
an Online Simulation Exercise
in Homeland Security Education
Ouanessa Boubsil
Interim Executive Director, Security Studies Laboratory
Outline
• Background University of Maryland University College
Homeland Security and Information Assurance Education
• Simulation Exercise
The village of San Luis Rey (SLR)
• Study Setting Methodology Study Tools • Results SLR Survey results
Course Evaluations results
The University of Maryland University College
• Founded in 1947 to provide education for military personnel and working adults
• Fully accredited institution of higher education headquartered in Adelphi, Maryland, USA
• Globally focused with more than 3,300 faculty members and 90,000 students worldwide
• 29 bachelor’s degrees, 19 masters degrees, a doctoral program and more than 80 certificates, most of them offered both online and face-to-face.
Homeland Security &
Information Assurance Programs
• Undergraduate Majors Homeland Security
Information Assurance
• Specializations within different Master of Science degrees Information Assurance
Homeland Security Management
• Undergraduate and Graduate Certificates in Information Assurance
Homeland Security Management
• Doctor of Management program Core course requirement Enterprise Continuity and Information Assurance
Graduate Enrollment
Gr aduate e nr ollm e nt in Infor m ation As s ur ance (IA) and Hom e land Se cur ity (HS) cour s e s
1999-2007 62 111 115 201 465 862 1295 1489 1751 0 200 400 600 800 1000 1200 1400 1600 1800 2000 FY 99 FY 00 FY 01 FY 02 FY 03 FY 04 FY 05 FY 06 FY 07 FY E n ro lle me n t
Undergraduate Enrollment
Unde rgr aduate e nrollm e nt in Infor m ation As s ur ance (IA) and Hom e land Se cur ity (HS) cours e s
1999-2006 351 579 725 820 1003 1197 1509 1829 0 2 0 0 4 0 0 6 0 0 8 0 0 10 0 0 12 0 0 14 0 0 16 0 0 18 0 0 2 0 0 0 FY 9 9 FY 0 0 FY 0 1 FY 0 2 FY 0 3 FY 0 4 FY 0 5 FY 0 6 F Y
The Simulation Exercise:
The Village of San Luis Rey (SLR)
• Multimedia scenario-based simulation, enhanced with news releases, online discussions, photo and schematic imagery, and audio elements to create a realistic experience
• A large database of sector reference material including maps, news articles, a library etc.
Setting
• An introductory graduate course in Homeland
Security-including infrastructure protection and issues in technical areas such as interconnectivity and interoperability
• A scenario-based simulation exercise -The ‘Village of San Luis Rey‘ (SLR)- by Teleologic
• Learners apply their learning of homeland security and critical infrastructure protection to try to prevent a multi-prong terrorist attack on the critical infrastructure sectors.
Project
• Class is divided into Homeland Security Strategy Teams (HSSTs) –4 or 5 students
• Each HSST is assigned to the protection of one of six critical infrastructure sectors (Banking, Electric Power, Water,
Healthcare, Transportation and Communication)
• HSSTs inventory, assess, analyze and submit a written report to the mayor of the village of SLR detailing their team’s findings and recommendations on how to protect the infrastructure.
Methodology
• 2 modules of the same course, taught by the same instructor, using the same textbooks and other instructional materials • 2 consecutive semesters (Fall 2005 and Spring 2006)
• Fall 2005 the control module used a group paper • Spring 2006 used the SLR simulation exercise
Study Tools
• Student Survey specifically designed for the use of SLR
• The generic end-of-semester course evaluations used for all courses taught at UMUC
• Course statistics measuring students participation and interactions
San Luis Rey (SLR) Survey
• Subjects 2 online sections of ITSM 620 – ‘Concepts in Homeland Security’
56 students in total • Survey
30 questions
41 completed surveys (77% response rate)
Areas Surveyed
• Personal information • Technology • SLR exercise content • Students learning • Students satisfaction • Team dynamics• Exercise organization and workload requirements • Students comments and suggestions
Survey Results: Strengths
• Login was easy• Links worked well
• Technology functioned well
• Sector information was mostly helpful • SLR helped in gaining knowledge
SLR helped in gaining knowledge
0% 10% 20% 30% 40% 50% 60% Per c ent a ge 1 2 3 4 5 Op in io nThe SLR exercise w as helpful in gaining understanding of critical infrastructure
concepts
1: S trongly Dis agree 2: Dis agree 3: Neutral
SLR was enjoyable
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Pe rc e n ta g e 1 2 3 4 5 OpinionI enjoyed the exercise
1: Strongly Disagree 2: Disagree 3: Neutral
SLR would attract students
0% 5% 10% 15% 20% 25% 30% 35% 40% Pe rc e n ta g e 1 2 3 4 5 OpinionI would take another course that uses the SLR exercise
1: Strongly Disagree 2: Disagree 3: Neutral 4: Agree 5: Strongly Agree
Improvement areas
• Library information• Maps
• Staged scenarios
• Time period too short
Comments Common Themes
• Some felt it added depth to the course• Some wanted to be in more than one sector • Directions were too broad
• Time to complete the exercise was too short • Too much information
Student Course Evaluation
Evaluation Item/Module on a scale of 1 to 5 0509(without simulation) 0602(with the simulation) Course Overall 4.05 4.20 Interaction 3.90 4.18 Pedagogy 4.15 4.46 Course Objectives 3.94 3.99Students Interactions
Item/Module 0509(without simulation) 0602(with the simulation) Conference postings (Number of words) 77428 263632 70858 243338 Study Group postings(number of words)
20644 42343
Students Comments 0509 (without the SLR)
Strongest features of the course
Reading the material (2/6)
Discussion and Interaction (2/6)
Recommendations to improve the course
• Less paper writing assignments
• Adapt course to include response to natural disasters and illnesses
Students Comments 0602 (with the SLR)
Strongest features of the course
• Teacher developed a really cool project which helped us learn a lot
• Strongest feature was class discussion(2) • Most invigorating course I have taken • SLR was the best
• Enjoyed interaction with other class members although we don’t always see eye to eye
Positive responses when prompted for
areas for improvement
Recommendations to improve the course
• I can’t think of any at the moment - I am very pleased • It is a solid built course
• Should be taught face to face (2)
• Needs a sequel– as for this course it was excellent. There are no further recommendations.
• Although the topic for the group project was a good idea, …I would not change the scenario the week the project is due.
Areas for further exploration
• Correlation between satisfaction and select variables from the survey such as sector assigned, grade, etc.
• Focus on the cognitive gains by comparing samples of the written assignments
• Group dynamics and effects on student performance and satisfaction ( size and homogeneity)
• Explore the Return On Investment (ROI) factors--Impact on enrollment and retention
