13.1. STI develops and maintains a portfolio of distributed learning environments directed toward maintaining an equivalent learning experience across all modalities.
STI students are adult learners and most are already employed in high-pressure careers.1 Many are key technical information security professionals for their employers and therefore are first responders when cyber intrusions are discovered, new systems must be assessed for security flaws before being deployed, or partnerships or acquisitions are being considered and a rapid security assessment is required. In other words, STI students are not in complete control of their time and they may not have the flexibility to be away from their offices for a full week to attend a Residential Institute for face-to-face live learning. Through SANS, STI is able to offer multiple distance learning options that enable our students to progress toward achieving their program learning outcomes even when they cannot attend the Residential Institutes.2 The multiple
distance learning options are designed to provide learning and assessment experiences equivalent to what students get in the Residential Institutes. In some cases, as the evidence below suggests, the distance learning options offer academic benefits that may be superior to those of the live classroom experience.
STI Offers Two Distributed Learning Modalities Synchronous “Live at a Distance:” vLive & Simulcast
vLive is a synchronous delivery environment with an asynchronous component designed to serve adult learners who prefer a classroom-style approach to learning, but cannot afford or do not have the time to travel.3 With the assistance of virtual moderators, faculty members lecture and respond to questions, assist students in exercises, and otherwise engage them during lectures at scheduled times on a regular basis over an extended period of time (e.g., Tuesday/Thursday evenings from 7-10 p.m.). Each lecture is also recorded, including student interaction and discussions. Students who miss a particular lecture can view the material at a later time.
Additionally, recording allows any student to refer back to the archived lecture for as long as six months after the course is completed to facilitate review for preparing for the associated GIAC examinations. The lecture material is supplemented in almost all cases by a digital laboratory environment.
1 Appendix 12: Student Enrollment Profile 2007-2012.
2 http://www.sans.edu/academics/course-delivery-options
3 Example of vLive course available in on-site Resource Room.
Demonstrations of the vLive experience may be freely accessed at
http://www.sans.org/vlive/demos.
Simulcast is the simultaneous multicasting over SANS’ vLive delivery system of a course while it is being taught at a Residential Institute. In addition to cameras and audio equipment in the room, a live moderator sits in class and represents Simulcast students, literally raising her/his hand when those Simulcast participants have a question, delivering the question as part of the live class, and ensuring the Simulcast participant is satisfied with the answer. This is particularly useful in the case of certain courses, like MGT 433, which rely on a higher degree of
interactivity between the student and faculty member, and for which a delayed view might be suboptimal.
OnDemand
OnDemand is an asynchronous digital learning environment that allows students to engage whenever and wherever they wish.4 A top-ranked instructor presents the material live and is recorded, slide-by-slide. That material is delivered in modules, corresponding to the segments of a day at a Residential Institute. OnDemand students thus have the opportunity to listen to the highly-rated teacher and watch the same slides they would have in a Residential Institute, but they have the added benefit of being able to stop the lecture and go back within a single slide, to a former slide, or even to a former chapter in order to hear something that might have gone by too quickly. Further, at the completion of each module, OnDemand delivers self-assessment questions to reinforce learning objectives. The student consumes the course modules in order, and the lectures are supplemented in almost all cases by a digital laboratory environment containing exercises unique to each course. OnDemand courses are also supported by a “virtual mentor” who responds to questions within 24 hours. When necessary, the virtual mentor accesses STI faculty for additional support. The educational delivery software for OnDemand was
developed and is maintained by SANS.
Students Are Assessed Using Identical Testing and Exercises, Regardless of Delivery
Modality, and GIAC Scores Indicate the Learning Outcomes are the Same Across Modalities. The learning objectives, content, and student testing are identical for all STI students regardless of whether the student attends a Residential Institute or uses OnDemand or vLive training. To help students earn the high scores STI requires on the globally-standardized GIAC exams, the course coverage must be equally comprehensive across all modalities. Each instructor employs his or her own examples and experiences to make the material memorable and to give the students confidence that what they are learning is, in fact, effective in improving information
4 A demonstration of an OnDemand course will be available in on-site Resource Room.
security. But regardless of modality or instructor, GIAC test scores and student evaluations provide substantial evidence that the three modalities are providing equivalent value.
As discussed in Section 7, and further in subsections 11.4, 14.3 and 14.4, GIAC exam scores are reviewed quarterly for each STI student, and no particular patterns have been noted regarding the relative impact of each course delivery modality. This judgment was due, however, to a series of individual quarterly judgments, and was not the result of an assessment over time. As part of our self-study process, we endeavored to evaluate exam performance data longitudinally in order to answer the question, “Does the choice of course delivery modality have an impact on measures of students’ learning outcomes?” In other words, can we be confident that our distance learning options offer students an educational experience as effective as the in-person, residential institute experience?
To answer this question, during self-study we analyzed 450+ GIAC exam scores received by STI students, relative to the course modality they had taken prior to the exam. The results of this analysis, shown in Figure 13.1, indicate that the modality of the course has no significant relative impact, on average, on the grades STI students achieve on subsequent GIAC exams.
Figure 13.1: GIAC Exam Results for STI Students, By Modality of Course Delivery
As shown, the average GIAC exam scores achieved by students who took their courses in-person at a Residential Institute, through OnDemand, and through vLive were 88.4, 89.4, and 87.7, respectively. 5 The students who took their courses through our OnDemand system actually 5 Figure 13.1 is based on 455 GIAC exams, of which 327, 104, and 24 of the exams are associated with students who took the associated course through a Live/Residential Institute, OnDemand, or vLive experience, respectively. An analysis of these data confirmed that there are no statistically significant differences in the exam scores of
achieved slightly higher average scores, although these results are not statistically significant. We believe this is a result of both the quality of the delivery technology and process, as well as the fact that we typically employ our highest-rated faculty in the OnDemand versions of our courses.
These data represent a mix of all the certification exams and courses taken by STI students, but the absence of any significant impact of course modality on exam scores is also seen when analyzing individual exam results, albeit with fewer data points available for analysis, as shows in Figure 13.2:
Figure 13.2: Individual GIAC Exam Scores, by Delivery Modality Taken by Student
GIAC Associated
Exam Course Res Inst OnDemand vLive
GSEC SEC 401 Average Grade: 90.6 88.9 86.8
Count: 38 20 4
GCIA SEC 503 Average Grade: 87.0 86.8 n/a
Count: 40 12 n/a
GCIH SEC 504 Average Grade: 91.2 92.7 90.2
Count: 45 15 5
Delivery Modality
Quality of the Distance Learning Environment Is Continuously Assessed
STI maintains high standards for distance learning environments and holds it partner SANS to those same high standards. SANS has set up a quality-control and remediation program to ensure that distance learning is an equivalent experience to Residential Institute learning for its students. The quality of the distributed learning courses is assessed based on student feedback from their course evaluations posted (as a link) at the end of each section of the course. This information allows STI and SANS to assess and compare distance learning effectiveness module-by-module and target recommended improvements where they are most needed. In order to learn more about what could be improved, the SANS Quality Assurance Manager views every student evaluation and contacts each student who rates a course module below an established threshold or expresses a resolvable concern.6
students who took the relevant courses through OnDemand or vLive, relative to the scores of those who took them in-person at a Residential Institute.
6 S.E. 7.6: OnDemand and vLive Distributed Learning – Executive Analysis Presentation
Table 13.1 shows the average of all ratings given year-to-date to our OnDemand and vLive offerings by all SANS students, based on a 1-5 scale.
Table 13.1. Student Ratings by Category for vLive and OnDemand
Student Ratings by Category, Jan - May, 2013
vLive 2013 Avg
Course Content 4.70
Labs Ease of Use 4.61
Labs Value 4.73
Teaching Skill 4.75
OnDemand 2013 Avg
Course Content 4.41
Labs Ease of Use 4.24
Labs Value 4.38
Teaching Skill 4.52
As shown, the average ratings for all course components are very satisfactory on an absolute scale. As importantly, these numbers compare favorably to the ratings received at live conferences. For example, the average rating for Teaching Skill given to all Certified SANS Instructors at Network Security 2012, one of SANS’ largest Residential Institutes, was 4.67, slightly lower than the rating given for the vLive experiences (4.73) and only a little higher than the ratings given from the OnDemand experience (4.52). During 2013, vLive students have been asked to rate the overall value of their courses, and their average response of 4.70 was higher than the average response (4.52) for the mix of courses given at Network Security 2013.7 Lastly, 99.95% of vLive students asked during this period “Would you recommend this course?”
answered affirmatively.
All student contacts and requests for new features or changes are logged, and this information is shared with course authors, instructors, and STI staff. Additionally, the SANS Quality
Assurance Manager oversees quality-control checking of all courses prior to publication/posting for registration to ensure that they meet the standards that have been set before the course “goes live” for enrollment.
7 OnDemand students were not asked this question during the period, because another question on player quality had been substituted in its place.
This constant assessment process has led to many opportunities to improve the distance learning environments. Examples of some of the interventions and improvements in 2012 were shown in Table 7.9, repeated below:
Table 7.9. Actions Taken to Improve Distance Learning, 2012
Trigger Analysis Hypothesis Implementation Post-analysis
Browser technology update Discovered textbook versions out of sync Students dissatisfied with course Q&A
Lab scores lower in distributed than in traditional classes
Speakers too fast for vLive environment
Browser add-ins were examined
Worked with web team to examine order process
Review of course audio Q&A revealed faculty not consistently repeating questions Interview with students to understand
interaction with labs
Reviewed lectures of “fast speakers” in distance environment for confirmation
JAVA program caused issues
Multiple versions in inventory caused selection issue Proposing that multi- directional
microphones may solve problem Downloading to student’s variety of computers leads to variable operational issues
Some faculty unaware of needs for different pacing in streaming environment
V4 of OnDemand interface implemented Linked text version to course audio on ordering
Pending
Supplemental lab installation documentation being developed
Tips and tricks document prepared for vLive faculty training before first course
Student experience scores have increased Five months of registration monitoring showed no repeat of issue Pending Pending Under way
Understanding the Current Availability of Distance Learning Options and Potential Expansion Following Accreditation
As mentioned in several other sections of our self-study, distance learning options represent an important medium by which our students take courses, and use of this medium has been associated with enabling a faster pace of study untethered to the time or cost requirements associated with the attendance at a Residential Institute. If we desire to increase the average pace of student progress, it follows that we must assess the availability of these options for our students. Table 13.3 represents the availability of courses in the MSISE program through SANS’s vLive or OnDemand mechanisms.
Table 13.3. Distance Learning Options for MSISE Courses, Second Half of 2013
OnDemand? June July Aug Sep Oct Nov Dec Jan, 2014 Core/Required MGT 305 No SEC 401 Yes MGT 433 No SEC 503 Yes SEC 504 Yes MGT 525 No MGT 421 Yes MGT 566 Yes Electives Aud 507 Yes DEV 544 DEV 544 Nov 4 - 27 FOR 408 Yes FOR 526 FOR 610 Yes SEC 542 Yes SEC 560 Yes SEC 575 SEC 617 Yes SEC 642 Nov 11 - Dec18 Jun 24- Jul24 Jul 23 - Sep 3
vLive Course Scheduled
(Unavailable) (Unavailable) (Unavailable) (Unavailable) Jul 15 - Aug 7 Oct 7 - Nov 13 Dec 3 - Jan 16 Oct 14 - Nov 20 Dec 3 - Jan 23 Sep 24 - Oct 31 Aug 5 - Sep 11 Nov 5 - Dec 12 Nov 4 - Dec 11
Jul 8 - Aug 7 Dec 2 - Jan 15
Sep 2 - Oct 16
Analysis of Table 13.3 indicates that (1) all of the larger, 5-6 day core SEC (Security) courses are available at any time through the OnDemand service, or at least once over a period of six months through a vLive offering in the evenings, and (2) many other electives are available during the next six-month period, either via OnDemand or through a scheduled vLive course. However, the table also shows that SANS does not currently offer MGT 305, MGT 433, MGT 525, or MGT 421 in the vLive format, and of these, only MGT 421 is available OnDemand. This latter group of four courses represents six of the 34 credits required in the MSISE curriculum, half of which are solely MGT 525 (a three-credit course). While these courses represent only about a sixth of the total credits required, STI will work with SANS to increase the ease with which STI students can take these courses outside of Residential Institutes. (Not shown in the table is the fact that MGT 433 is offered via Simulcast at each of the largest five Residential Institutes, which eliminates a student’s need to travel but would still require the person to take a day off from work to sit in “live” at a distance.) Options include presenting these four courses through the OnDemand format or enrolling students after accreditation in cohorts of sufficient size to justify a dedicated vLive class or series of classes.
Attachment G-3: Instructor Tips for Distance Courses