COMPARING THE EFFICIENCY AND EFFECTIVENESS OF DIFFERENT LEARNING MEDIA DESIGNS FOR THE PREPARATION OF SKILLS TRAINING IN A BLENDED LEARNING CONTEXT

COMPARING THE EFFICIENCY AND EFFECTIVENESS OF

DIFFERENT LEARNING MEDIA DESIGNS FOR THE PREPARATION

OF SKILLS TRAINING IN A BLENDED LEARNING CONTEXT

N. Verliefde

, L. Stevens

, B. D’haenens

, A. Vermeyen

R. Kaat

, M. De Gendt

,

C. Van Caelenberg

, K. Stifkens

, S.Van Rampelberg

, T.Neijens, J. Van Den

Bossche

1

_{Association of the Catholique University of Leuven (BELGIUM)}

_{KAHO Sint-Lieven, Service for further education, Ghent (BELGIUM)}

3

_{KAHO Sint-Lieven, Service for educational support and development, Ghent (BELGIUM)}

_{KAHO Sint-Lieven, Aalst (BELGIUM}

₎

5

_{Technical Institute Sint Carolus, campus Waas, (BELGIUM)}

6

_{Hogeschool – Universiteit Brussel, Nursing Education, Brussels}

_(BELGIUM)

_{Catholic University College Sint Lieven, Nursing Education, Sint Niklaas}

_(BELGIUM

₎

8

_{Leuven Engineering and Sience Education Center (LESEC) (BELGIUM)}

Abstract

This practice based research on effectiveness and efficiency of learning media design is meant to provide an answer to the professional bachelor programs nursing of three affiliated university colleges of the Catholic University of Leuven and one associate degree program for the optimization of their blended learning approach for the practical skills component of the programs.

In particular we want to find the most optimal media design for students to prepare themselves independently for the practicum nursing skills. This replaces the demonstration of the skill that is normally done by the lecturer during the contact time of the practicum.

We compare three stratified groups of hundred students. Each group prepares themselves for the real practicum of the next day by means of a different online medium: a short video with limited narration (spoken information: “what”), a long video with elaborate narration (more detailed spoken information: “what”, “how”, “why”), and a multimedia interactive learning module with learner control.

We find supportive results for the decision to augment narration in the instruction videos from limited to elaborate narration. Integrating learning objects into a multi media module with learner control however, does not influence the learning outcomes/results (learning effectiveness) nor the time spent on the lab preparation assignment (learning efficiency).

The final aim is to decide on basis of a cost-effectiveness evaluation on the future media design to be used in both the pre-service and post-service nursing programs.

Keywords: learning effectiveness, media design, multimedia, blended learning, instructional media, practical skills education, nursing skills training, asynchronous learning

1

INTRODUCTION

The rise of new multimedia technologies for online learning, brings new opportunities for the practical components of educational programs to make the contact times during practica and skills lab more efficient. In the blended learning approach of the nursing programs involved in this project, the face-to face demonstration of the nursing skill is replaced by an online instruction video. This not only saves

time in se, but also allows students to better prepare themselves before coming to the skills lab, which allows them to spend more time practicing the skill with larger insight in the procedure of the skill. Blended learning is generally defined as an educational approach whereby face-to-face and online methods are combined [1]. In Napier’s [2] opinion ““Blended learning significantly reduces face-to-face instruction by incorporating rich, online learning experiences”. In our blended learning approach the focus is more on increasing the quality of the face-to-face instruction, rather than reducing it. The lecturer can shift his/her role from knowledge transfer to guidance of the learning process.

We investigate whether the online learning experience we offer by means of the instruction videos can be made “richer”. The central question is: which choice of media and media design can offer the best support to nursing students to prepare themselves optimal for the practicum session/ skills lab ? In a preliminary satisfaction questionnaire on the quality of the instruction videos used in the technical nursing skills course, the main wish from the students was more elaborate narration (spoken information) in the video and to align the video better with the steps of the written procedural guidelines of the technical nursing skills manual. This however contradicts with assumptions and believes of the lecturers on effectiveness of the media design:

- “If we do what students ask, explaining everything they need to know in the instruction video, they won’t look at their textbooks (technical manual) anymore !”

The focus from the service offering further education nursing programs is more on “attracting” clients and is therefore considering a multimedia design of attractive (inter)active learning modules with learner control. Here cost-effectiveness questions raise:

- “Is the time consuming production process of an interactive multimedia module worth the effort ? - “Is there an equivalent less time consuming way to add active learning activities to the passive watching of a video, by means of our learning management system?”.

In this comparative analysis we investigate the effect of the educational design of online instruction video and of an interactive multimedia module on the learning process as well as on the learning outcomes

2

OBJECTIVE

To be able to make an evidence based decision on which media design to opt for in future, we compare:

the effectiveness and efficiency of the use of different learning media designs for students to prepare themselves optimal for a practicum / skills lab.

3

RESEARCH QUESTIONS

Learning effectiveness

We are primarily interested in the learning effectiveness, the effects on the learning outcomes. Which media design realizes the largest level of recall and insight in the procedure of the practical nursing skill ? The two main research questions on effectiveness are:

1. Do elaborate self sufficient instruction films contribute to learning effectiveness as compared to short videos whereby one still has to read the written textbook or guidelines to understand the rationale behind certain steps of the procedure ?

2. Does a flexible interactive learning path of a multimedia module with integrated learning objects (text, video, self-evaluation questions with immediate feedback ) significantly enhance learning as compared to a linear structured instruction video offered separately from the other learning objects ? Learning efficiency

Is the time spent on the assignment in the group of learners that obtain the learning objectives in a sufficient way (effectiveness) less in any one of the media designs ?

Do students perceive the integrated learning module as more attractive than the short video or long video ?

4

EDUCATIONAL PRINCIPLES ON MEDIA DESIGN

In the following table we summarize the characteristics we take into consideration to improve the quality of the learning media design. They are derived from the highlighted wishes from our students as well as from the principles of the cognitive learning theory on multimedia from Mayer [3] and the Framework of Screenwriting principles for instructional video from Koumi [4].

Table 1: characteristics of the learning media design Media design 1: short

instruction video

Media design 2: long instruction video

Media design 3: Interactive multimedia (learning module)

RESEARCH QUESTION 1

limited narration elaborate narration

RESEARCH QUESTION 2

Separate learning objects: Audio-visual: video (short) + text (written guidelines) + self evaluation questions added in LMS (blackboard)

Separate learning objects: Audio-visual: video (long) + Text (written guidelines)

+ Self evaluation questions (SEQ) added in LMS (blackboard)

Integrated learning objects Multimedia: video (long video), text, SEQ, with segmentation..

Linear learning path Linear learning path Flexible learning path ,with

learner control. (Inter)activity: SEQ with

delayed corrective feedback

(inter)activity: SEQ, with delayed corrective feedback

Interactivity: SEQ with immediate corrective feedback.

Research queston 1: elaborating narration  effect on learning ?

Whether an elaborate video contributes to learning effectiveness as compared to a short video with limited narration, we need to know how much narration in the video is beneficial for learning and what should the content of the narration be ?

In the preliminary satisfaction questionnaire and interview students suggest:

- more narration explaining “what is done, how it is done and why it is done”.

In the current video design the narration (spoken information) accompanying the moving images, is often limited to “what is done”. Lecturers limit the amount of information deliberately, because they are afraid students won’t look at their textbook / written procedural guidelines anymore if they explain too much in the video.

The narration in the elaborate video contains:

-“What”: Labeling the steps of the skill  offers structure, matching the sequence to the written guidelines of the skill.

-“How”: Stressing specific attention points on how to carry out a (sub)step of the skill or on how to make the right choice of materials

e.g. “See to it that you stretch the skin of the injection place before injecting the needle !” e.g. “Watch out: the higher the gauge the thinner the needle !”

-“Why”: explaining the importance of the attention point, explaining the positive consequences when carried out correctly and the negative when it is omitted.

e.g. “You chose a short needle when the patient is skinny, otherwise you might hit the bone.” Is what the students wish, being supported by Koumi’s theoretical framework of video design and screenwriting principles [4]? Indeed Koumi explains under his constructivist principle: “words should not duplicate pictures, they should add something extra”. But under the other hand he warns for overload.(Koumi’s 6.3 principle).

If one is aware of the importance of a step in the procedure, has therefore been able to link the new knowledge to an existing knowledge construct in the brain, one is less likely to forget it. (constructivist principle). But if we overload, the student can not pay attention to the sequence of the steps anymore

Research question 2, integrating learning objects, learner control, adding interactivity  effect on learning ?

Whether a flexible interactive multimedia module with integrated learning objects significantly enhances learning as compared to a linear structured instruction video offered separately from the other learning objects, we make the following assumptions:

- Integrated learning objects; Mayer’s principles on multimedia design:

Multi media principle: people learn better from words (written and spoken) and pictures than from words only.

Segmenting: presenting in learner-paced segments

Spacial contiguity: people learn better when corresponding graphics and printed text are placed near each other.

 In the learning module of media design 3, we cut up the written procedural guidelines into sections

(sub steps of the skill) which we put in text frames together with the fragment of the video illustrating this sub step. We still provide the full video version as well in one of the first frames of the learning module.

However reading and listening to information at the same time does not go together (redundancy principle of Mayer).

 Students can chose which one to look at first, text or video, and switch easily between the two. - Flexible navigation, “learner control” of the learning path is more effective because the learner can control the learning path according to own preferences and in line with the learning style. He can chose for: linear or non-linear; holistic versus analytical (long video, short fragments), sequence of content by means of menu-bar in the left column, sequence of type of learning object: text / video,… However research on the Aptitude Treatment Interaction from Cronbach & Snow [5] show that not all learners profit from learner control. Snow states that low achievers and anxious or conformist type of learners tend to learn better in highly structured instructional environments. The high achievers and independent students profit from “learner control”, which is confirmed by Clark & Mayer [6].

 Our student group, Flemish professional bachelor nursing students of the Association of the

Catholic University of Leuven, are experienced by their lectures as being “docile, not critical, with a “tell us what to do attitude”. In general nurses conform to carrying out tasks according to protocols and decided by the doctors. Therefore it is not excluded that they might profit more from linear, structured instruction, than from free learner control.

Another aspect which might undo the advantage of flexible navigation is the typical sequential nature of the skill to be learnt. The learning objectives are: to know and understand the necessary steps of a practical nursing skill and their proper sequence in the procedure of the total skill. - (Inter)active learning activities are more effective than instruction design whereby the learner only passively listens or watches a video. (Learning Piramid of Bales, constructivist learning theory)

 “Interactive learning activities” are integrated in our learning module in the form of corrective

feedback on “test yourself” questions used during the process of learning. The questions help to consolidate the insights learnt from watching the demonstration video and reading the textual

information. Questions in the learning module are integrated and attached to small learning sections/units.(cfr. Segmenting principle of Mayer) Students have control over which question they want to answer first, and get immediate feedback on their answer. With normal video (media design 1 & 2) we only got the possibility to add the “test yourself” questions with feedback as a separate learning object in the LMS (Blackboard). Students get only the possibility to answer the questions one after the other and they get the correct answers at the end of the test. We add the “Test yourself” questions also to the two other learning media designs in order to keep the learning activities as close to equal as possible.

Clarks [7] states that “media in itself do not influence learning”, but the type of learning activity in the instructional design. Therefore if we want to isolate the effect of the media design itself, we have to keep the learning activities as close to equal as possible.

5

RESEARCH METHODOLOGY

In order to isolate the influence of the media design (MD) on learning, we compare three experimental groups of hundred students that only differ in the type of media design used. (see table 1, p.4)

Experimental group 1: media design 1 (MD1): short video Experimental group 2: media design 2 (MD2): elaborate video Experimental group 3: media design 3 (MD3) learning module

To rule out other influencing factors of the characteristics of the learning groups, we divide the students of the four participating institutions equally over the three experimental groups and we stratify them also on: type of enrolment (distance education or traditional education), prior work experience, previous education, gender, age and level of ability. The level of ability is defined by means of the study results of the previous semester (1st years, 1st semester) on the same practical nursing course. We use the normal distribution curve of each of the institutions to identify the high achievers, medium and low achievers.

For research question 1: we compare MD1 with MD2, that only differ in elaboration of narration. For research question 2: we compare MD2 with MD3, whereby MD3 adds: integration of (segmented) learning objects, learner control and immediate character of the corrective feedback on self evaluation questions.

In order to make an educational policy decision on the future use of media design based on the results of this research, the experimental learning task needs to be representative for the real education situation. Therefore we incorporate the experimental lab preparation assignment into the real education program. The day before the real practicum on the chosen skill “intramuscular injection” takes place, the students get two hours to prepare themselves to achieve the necessary insights in the procedure of the skill and to remember the steps in the right sequence.

Keeping the research design both experimentally controlled as well as representative for the real educational situation is very challenging. We’ve involved all the lecturers in order to standardise the prior knowledge on the specific topic as well as on the related preparatory ones.

As evaluation method to measure the effectiveness of the media designs, we conduct a written test (mainly multiple choice) at the beginning of the practicum, the next day. The test scores reflect the recall and insights in the steps of the procedure that are necessary to apply the skill correctly. A team of five subject lecturers identified these crucial insights, based on an analysis of frequent misconceptions an forgotten sub steps of the procedure observed in the students’ performance at the practical exams. It is also the same group of subject specialists who has decided on the content an narration in the video and the consolidating “test yourself” questions.

The learning efficiency is measured by means of an online questionnaire that students had to fill in at the end of the lab preparation assignment. We compare the indicated time spent on the assignment and analyse if there is a difference between the three experimental groups?

The online questionnaire also asks other details of the learning process: the use of the traditional written guidelines, the use of the questions with corrective feedback, the attractiveness of the media design, the pleasant way of learning, the perceived effectiveness of the learning media.

6

RESULTS

6.1 Learning effectiveness

6.1.1 Research question 1: Comparing the learning results of media design 1, video

with limited narration with media design 2, video with elaborate narration

The comparison (by means of a T-test) of the total test scores of the media design 1 group, (video with limited narration) with the media design 2 group, (video with elaborate narration) points out a significant difference (p=0,003) in favour of the elaborate video.

The difference in mean score of the written test results is however only 0,5/10. Figure 1 shows the distribution of the test scores for the three types of media design.

Fig 1 Distribution of total written test score by media design

We can conclude that the students’ wish for more elaborate narration in the video pays off. Moreover do our analyses of the use of the written procedural guidelines (textbook) contradict the fear of the lectures that when using elaborate videos the students would not use the written guidelines anymore. In figure 2 we can see that there is almost no difference in percentage of respondents of the different media design groups in the use of the written procedural guidelines. 76% of students of the short video group say they have used the written guidelines, while also 74% of students of the elaborate video group say to have used them.

Fig 2 Use of written procedural guidelines by media design groups

6.1.2 Research question 2: Comparing the learning results from media design 2,

(elaborate video), with media design 3, (learning module).

The group of learners using the learning module does not get higher results than the group using the elaborate video (media design 2). On the contrary, in table 2 we can see that the mean test score shows slightly lower results for the learning module group. The difference is so small, that it should be neglected.

Table 2 Mean test scores by media design

1: Short video 2: Elaborate video 3: Learning module Total test score

Mean test score 5,59 6,08 5,94 / 10

Our findings correspond with the findings of Snow and Mayer, that learner control is not necessary beneficial for all learners. It would be worth to further examine if the lecturers’ perceptions on the characteristics of this learner group of nurses, as being in general “conformist”, is being confirmed by empirical evidence.

In general overlooking or bypassing information that might still be important, happens more with a flexible learning path, than with a linear structured learning path. [8] The results from our learning process questionnaire show in fig. 3 that students from the learning module (media design 3) bypass some “test yourself” questions (only 29% has made them all), while the other groups (media design 1 and 2) make significantly more “all the questions” (group 1: 79%, group 2: 67%).

Fig 3: Use of “test yourself” questions grouped by media design

0% 20% 40% 60% 80% 100%

not used used

P er cen tag e o f re s p o n d e n ts

Use of written procedural guidelines

short video elaborate video learning module 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

none some almost all all

p e rc e n ta g e o f r e s p o n d e n ts

test yourself questions

short video elaborate video learning module

6.2 Learning efficiency

Figure 4 shows the analyses of study time spent by the different media design groups. There is no consistent difference found between the three experimental groups in time spent on the assignment.

Fig4 Study time spent grouped by media design

We also conducted analyses on the subgroups who obtained the learning outcomes in a sufficient way (test score 5/10 or higher) and the subgroup whose learning result was optimal (score 7/10 or higher) in an optimal way. No particular media design causes students to go faster through a task in any of these subgroups of learners.

6.3 Motivational aspects of learning: attractiveness

For the analyses of perception on factors that can influence motivation for learning, we used a 6 point likert scale

.

6.3.1 Attractiveness

Students found all the three media designs equally attractive.

Fig 5 Perceived attractiveness grouped by media design

0% 5% 10% 15% 20% 25% 30% 35% <20' 20'-30' 30'-45' 45'-60' 60'-90' >90' Per cen ta ge o f r es po nd en ts

Study time spent

short video elaborate video learning module 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 1 2 3 4 5 6 Per cen ta ge o f r es po nd en ts Attractive (Likert) short video elaborate video learning module

6.3.2 Pleasant way of learning

In figure 6 we see that the way of learning by means of the learning module gets less respondents in the category 5 “very pleasant” (10% to 17% less respondents). The open questions pointed out that the additional short video clips were sometimes too short. It was too scattered.

Fig 6 Appreciation of pleasant way of learning grouped by media design

6.3.3 Perceived effectiveness

In figure 7 we see a similar tendence as in figure 6/. Category 5 “very effective” has 10 % less respondents for the learning module (media design3). The experimental group 3 perceives the learning module a bit less effective than the other two experimental groups perceive the effectiveness of their media design in their preparation for skills lab or practicum.

Fig 7. Perceived effectiveness grouped by media design

0% 10% 20% 30% 40% 50% 60% 1 2 3 4 5 6 Per cen ta ge o f r es po nt en ts

pleasant way of learning (Likert)

short video elaborate video learning module 0% 5% 10% 15% 20% 25% 30% 35% 40% 1 2 3 4 5 6 P e rc e n ta g e o f c o rr e s p o n d e n ts perceived effectiveness short video elaborate video learning module

7

CONCLUSION

The comparison of the effectiveness of the three different learning media designs shows one very significant difference. Augmenting the amount of narration in the instructional video from limited, only saying “what is done” to explaining “what”, “how precisely” and “why”, has a positive effect on learning. The group of students who prepared themselves for the practicum by means of the elaborate video, get higher results on the assessment of recall and crucial insights in the procedure of the nursing skill. The fear of lecturers, that when using an elaborate instruction video, the written procedural guidelines would not be used anymore, therefore important insights might be lost, is contradicted by our research findings. This means that the wish of the students, to add more spoken content to the videos is justified.

The time consuming production process of an integrated multimedia module with learner control on the other hand, does not seem to be worth the effort for this group of nursing students. Students who prepare themselves by means of the learning module do not show better results, on the contrary slightly - however to be neglected- lower scores.

The equivalent less time consuming way to add active learning activities to the passive watching of a video, by adding questions with corrective feedback in the Electronic Learning Management System (ELMS, Blackboard) pays off. On the other hand, the advantage of integrating questions with corrective feedback in the learning module is questionable. It is observed that linking the “Test Yourself” to smaller learning sections, and giving the learner more choice which questions in which sequence to answer, results in bypassing important questions. Skipping questions happens significantly less when using the linear constructed test forms in the ELMS.

The integrated learning module not only fails to score better on effectiveness than the other two media designs, there are no significant differences found either in time spent on task (learning efficiency), nor on perceived attractiveness of the media design. The perceptions on way of learning, and the effectiveness of the learning module even scores lower for the category 5 of the 6 point likert scale. Further research will be carried out on the effect of the specific media design factors by comparing the results on specific written test questions that are linked to content presented in different ways. The impact of factors to be further examined are: signalling by means of textboxes in the video, structuring and highlighting sequence by means of title cues in the video.

Also possible improvements to the design of the learning module will be further analysed, using the remarks of attractiveness and effectiveness obtained in the questionnaire.

REFERENCES

[1] 7th Annual Sloan Consortium Blended Learning Conference and Workshop in April 2010

[2] Napier N.P., Transitioning to Blended Learning: understanding Student and Faculty Perceptions, Journal of Asynchronous Learning Networks, A publication of the Sloan Consortium, ISSN ISSN 1939-5256, volume 15, issue 1 – February 2011.

[3] Mayer R.E. (2009), Multimedia learning, Cambridge University Press, 2nd edition, ISBN 978-0-521-73535-3.

[4] Koumi J.(2009), instructivist exposition with constructivist learning opportunities, Educational Media Production Training, United Kingdom

[5] Snow, R. (1989). Aptitude-Treatment Interaction as a framework for research on individual differences in learning. In P. Ackerman, R.J. Sternberg, & R. Glaser (ed.), Learning and Individual Differences. New York: W.H. Freeman.

[6] Clark R.C. & Mayer R.E., e-learning and the science of instruction: proven guidelines for consumers and designers of multimedia learning, Pfeiffer, ISBN 978-0-7879-8683-4.

[7] Clark, R.E. (1994), Media will never influence learning. Educational Technology Research and Development, 42, 21-30