The three aspects of story described above were the focus of our evaluation methodology. Our methods included a questionnaire and a structured interview. The design of each question was driven by our predictions about how story impacts learners. A group of twenty four consultants completed the course. Each of these participated in the structured interview following course completion. Seventeen of the learners had the opportunity to answer the questionnaire items. Some items were presented only once while others were presented after each of the five course modules. Thus the range of questionnaire responses varied between seventeen and eighty-five.
We coded the responses using the framework in Table 1. The three rows represent the type of content in the responses, ranging from content only, integrated content and story, and story only. The columns represent the degree of detail ranging from general to specific to personal experience.
General Specific Personal Experience
Content Only 1.1 Mention of content at a conceptual level - no use of business process terminology
1.2 Mention specific reccomendations/issues or use of specific business process terminology
1.3 Relates business process content to personal experience.
Content and
Story 2.1 Mention of content atconceptual level with reference to the
engagement genre
2.2 Integration of both business process content details and story details
2.3 Related story and business process content details to personal experience. Story Only 3.1 Comments about
engagement genre only
3.2 Details of the story wirthout mention of business process content
3.3 Comments story events and characters related to personal experience.
Table 1. Response Analysis Framework
Memory Structure
Two components of memory structure were considered in our analysis. First, the BPO course used a client engagement genre which should have given the learner a familiar context in which to structure new experiences. To assess the effect of genre, the learners were asked directly and indirectly to rate the familiarity of the
simulated client engagement environment.
On a scale of 1 to 5 (1= not at all and 5= very much so), was this activity realistic compared to problems arising on an actual engagement?
The average rating was 3.93 based on the fifty eight completed samples collected after each course module by the twenty four learners. This finding confirms that the client engagement genre was communicated clearly in the course.
Learners were also asked to compare their experience in the course with other experiences.
On a scale of 1 to5 (1= Not at all and 5 Very similar), how similar was this course compared to: o Printed self-study experiences o A videogame
o Classroom experiences o Real project experiences
o Other computer-based training o Time with an industry expert o A novel
Learners consistently rated this experience to be most like real project work (mean = 3.86) which is consistent with the predicted effects of the client engagement genre. “Other computer based training” was ranked second most similar to the BPO course (mean 3.42) which also follows the original stated design assumptions. The experience was also rated to be highly similar to time with an industry expert (mean = 3.28). The course was rated least like printed self-study experiences (mean = 2.35) or a classroom experiences (mean = 2.50). Second, to assess the memorability of the business process content learned in the course we asked questions using cues for recall that varied in story detail. For example, the question: "What did you do when Synchro's LMM-MAX laptop experienced serious quality problems?" was asked to trigger detailed recall of the newly acquired script for process analysis. The question " What did you learn about how to help a company solve its quality problems?" was asked to see if less-detailed recall of the content would result. The free form responses were coded according to the content and level of detail. (Table 1.) Their answer types are presented following each question.
a.What specific things did you do to help Syncro solve their quality problem with the LLM MAX? b. What did you learn about how to help a company solve its quality problems?
Over 80% of the story-cued responses (a) were coded as 2.2, integrating details of both story and content. The general-cued responses (b) were evenly split between cells 1.1 and 1.2, showing less detailed recall of content. This result suggests that the both story and content details were encoded together in memory and that the story was an effective cue for recall of detailed content. One learner, when given the general question, talked for a while at the general content level (1.1), but continued .. ”I’m trying to remember some of the aspects of the exercises and what are the major problems we’ve all recognized....It’s very difficult to go back unless you can put it in some sort of context. I’m trying to remember the events of Quality [the name of the module in question] (pause)” The student then proceeded to generate a more detailed and integrated response (2.2). This suggests that the story served as an effective structure for storage and retrieval of content.
What was the most memorable or interesting thing about Synchro?
40% of the responses were coded as 2.2, integrating details of both story and content. Another 20% of the responses were coded 2.3, relating story and content details to personal experience. The remaining responses were spread out evenly across the other cells. This result is consistent with the other memory responses in that both story and content details were recalled together. The 20% which related the story and content to their experience indicate successful use of genre, as well as integrated learning of new content.
Taken together, the above results show that story in Business Processes Overview helped structure learners’ memory of the course content to facilitate detailed recall and integration with personal experience. Our data reveal that both the genre and story details are encoded by learners. We believe this result is important since genre represent a learning vehicle not typically exploited in computer-based education. When a learner
experiences any form of training, he or she primarily perceives the “training” genre and formulates expectations about the experience (e.g., “I can fail without significant risk,” or “This is not work”). Our data suggest that story can introduce additional genres to which learners are sensitive and enable them to form expectations for learning. The finding that this course was most similar to a real client engagement is evidence that the engagement genre was able to influence learner expectations and provide strong context for learning new business analysis behaviors.
Learner Motivation
In our research, we considered several aspects of motivation as part of the story experience. First, to assess the impact of story on learner motivation, we recorded subjective reports of interest after each module of the course. Second, we assessed the extent to which story had created expectations in the learner for what would happen next.
Please indicate how interested you are in the issues raised in this course (1= not interested, 10 = very interested. The average rating of 8.93 directly supports the hypothesis that the story sustained high learner interest.
Considering what you have learned up to this point about Synchro, what do you think will happen next?
Each response was coded according to the criteria presented in Table 1. Over 70% of the responses fell in the detailed categories (1.2, 2.2, 3.2). That learners were able to make predictions with supporting rationale provides evidence of their interest and attention for the sequence of activities and events.
The motivation results confirm that the story helped the learners maintain interest in the course over a sustained period of time. One learner revealed her excitement about the story in these words:
"The changes we are making to Synchro are pretty sweeping. I mean we are changing a lot of the things they are doing and for the purpose of the training I think it is necessary because you can concentrate on one area.... I think the storyline is very feasible and actually I'm very curious to know what happens in the end!"
These and similar comments suggest that learners were truly engaged in the plight of Synchro and were motivated to see the story to its end.
We embedded the culture of a typical products company with characters and typical inter-departmental dynamics to convey the culture of a functionally oriented company. To assess how well these messages were learned, we asked the following questions:
What cultural messages did you perceive either about Synchro or about Andersen from the course?
65% of the reponses were coded as 1.2, containing both content and story details. Another 29% were coded as 1.3, relating details to personal experience. This indicates that the cultural messages embedded in the story were recalled in detail by most learners, with some readily relating the messages to their own experience.
When you think about Synchro, do any of the personalities who work there come to mind?
All of the respondents recalled at least one character from the story. Half of the respondents were able to recall two or more characters.
The story proved to be an effective strategy for conveying the intended cultural messages helping learners relate the content to their experience.
Overall Impact of Story
We completed our assessment of the efficacy of story by simply asking learners what value, if any, they found in the story components of the course. All learners responded that the story was valuable in some way. 75% of the responses mentioned “maintaining interest” as valued; 67% of the responses mentioned content messages as valued, and 25% of the reponses related the overall story to their personal experience.
Conclusion
While it may be obvious that story is a desireable instructional strategy, it is important to understand the effects of story in the same terms we understand other strategies. The Business Processes Overview course is an attempt at systematically using tools from the discipline of entertainment -- action, character, conflict, and genre -- to enhance the learning experience. This study of the Business Processes Overview course and its effects on learners shows that story can add value to the learning experience in terms of motivation and memory for content at both detailed and general levels. Our story was intentionally grounded in both the course content and the learner’s experiece with a familiar genre. The intentional manipulation of genre in an educational context represents a robust vehicle to facillitate learning new behavior. Overall, story created a learning experience that integrated motivation, detailed content, and cultural content messages.
We have also introduced a framework for evaluating the effects of story on memory, outlining a progression from general content recall to detailed integration of story and content to integration with personal experience. Future research will focus on refining this framework and applying it to longitudinal studies of memory to assess the long-term effects of story on learning.