AIS Electronic Library (AISeL)
AIS Electronic Library (AISeL)
ICIS 2019 Proceedings Digital Learning Environment and Future IS Curriculum
Challenges in Switching to Blended Learning Environments: An
Challenges in Switching to Blended Learning Environments: An
Analysis of Students’ Attitudes and Performance
Analysis of Students’ Attitudes and Performance
Bernd SchenkUniversity of Liechtenstein, [email protected] Luiza Hoxhaj
University of Liechtenstein, [email protected]
Follow this and additional works at: https://aisel.aisnet.org/icis2019
Schenk, Bernd and Hoxhaj, Luiza, "Challenges in Switching to Blended Learning Environments: An Analysis of Students’ Attitudes and Performance" (2019). ICIS 2019 Proceedings. 5.
https://aisel.aisnet.org/icis2019/learning_environ/learning_environ/5
This material is brought to you by the International Conference on Information Systems (ICIS) at AIS Electronic Library (AISeL). It has been accepted for inclusion in ICIS 2019 Proceedings by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact [email protected].
Challenges in Switching to Blended Learning
Environments: An Analysis of Students’
Attitudes and Performance
Completed Research Paper
Bernd Schenk
University of Liechtenstein
9490 Vaduz, Liechtenstein
[email protected]
Luiza Hoxhaj
University of Liechtenstein
9490 Vaduz, Liechtenstein
[email protected]
Abstract
Blended learning has been promoted as a promising way to create a rich learning experience that uses students’ rising affinity for online media consumption and newly available technologies to improve students’ learning success. A mixed-method approach is used to investigate students’ attitudes toward and perceptions of blended learning in higher education and to evaluate changes in overall satisfaction and exam performance induced by blended learning. The data used is available for most courses taught in higher education institutions and can be used for the continuous improvement of blended learning setups. Although students were open to the idea of including blended learning methods, results indicate that accompanying measures are of paramount importance to achieve positive effects. Intrinsic motivation must be explicitly created in a blended learning environment to ensure learning success. The findings suggest that introducing technologies into learning environments requires a gradual transition between classical and blended learning methods.
Keywords: Blended Learning, student perception, student performance, mixed-method approach
Introduction
With the advancements in technology and the popularity of social media, special attention has been paid to the digitization of higher education. Many universities have increased the number of courses offered online and are gradually substituting the traditional approach to learning with them (Nguyen, 2011). The history of technological advancement in higher education has gone through three generational changes (So and Brush, 2008).
In the first generation, students learned with the traditional face-to-face approach where the instructor lectured in person and in real time. The second generation was characterized by the introduction of computer-based and online classes in response to the rapid development of the Internet. The third and current generation is called blended learning (So and Brush, 2008), which is a mix of traditional and e-learning approaches that can enhance the e-learning process, especially in higher education, by combining the benefits of both traditional and computer-based approaches (Dang, Zhang, Ravindran, and Osmonbekov, 2016). Incorporating technology into higher education attracts more students because it addresses issues related to the time required for higher education and access to classes in terms of physical distance (Aristovnik, Tomazevic, Kerzic, and Umek, 2017). Blended learning provides students
with a flexible approach to learning in terms of times and places, with some online sessions retaining face-to-face contact (O'Connor, Mortimer, and Bond, 2011).
Several studies have shown the positive aspects of the blended learning approach from the students’ and instructors’ perspectives (Ozkan and Koseler, 2009; Schaber, Wilcox, Whiteside, Marsh, and Brooks, 2010), including increased student engagement through social media, more flexibility, increased student performance, the higher attractiveness of courses, decreased cost of delivering educational materials, decreased dropout rates, and better usage of in-class time (Garrison and Kanuka, 2004; Kwak, Menezes, and Sherwood, 2015).
Besides the positive effects, studies have also shown that blended learning has several issues that have, in some cases, decreased student performance (Allen, Bourhis, Burrell, and Mabry, 2002; Kwak et al., 2015; O'Connor et al., 2011; Parkinson, Greene, Kim, and Marioni, 2003). As Donnelly (2010) observed, the implementation of blended learning has many challenges because the variety of media and technological advancements that are shaping the educational sector require a high degree of expertise in these topics on the side of the educators.
Universities and professors struggle when switching to blended learning settings for many different reasons. Professors need to keep up to date with current technologies to design a setting that is suitable for their needs. Furthermore, the role of the instructor switches from that of a content provider to one of curators of content orchestrated from different sources.
The development of technologies that are available to higher education has been rapid. Although blended learning has been introduced in educational institutions for some time, few research studies have evaluated students’ perceptions of this approach and their resulting performance. Based on the literature review conducted, none of the studies in this area have provided an integrated approach to analyzing students’ attitudes (formed by perception and motivation) and performance in blended learning settings. This study uses a mixed-method approach to investigate how various learning technologies and the way they are implemented in blended learning environments influence students’ attitudes and course performance. The study relies on existing data that are available for most of the courses taught in higher education institutions and evaluates course modifications by comparing the traditional version of a course with the blended learning version. The results of applying this mixed-method approach in the 2018 spring semester provide important insights into the challenges of implementing blended learning. The paper closes with implications and recommendations for onboarding student cohorts to blended learning endeavors.
Blended Learning
The term “blended learning” refers to a hybrid learning approach that combines traditional face-to-face education and online learning (Dang et al., 2016; Heinze and Procter, 2004; O’Connor et al., 2011). This learning method was developed as a result of advancements in communication and information technology with the aim of enhancing higher education. Many countries are now including IT and other innovative methods in their educational institutions, resulting in benefits that include quality improvements, flexible learning, and lower costs (Chiang, Wuttke, Knauf, Sun, and Tso, 2009). Eighty-eight percent of schools in the USA have access to the Internet (Herold, 2017), which is an important precondition for blended learning.
Although the ability to use the Internet is equally important for all age groups, its importance differs when it comes to social networking. Eighty-eight percent of European Internet users between the ages of 16 and 24 engage in social networking (Eurostat, 2019). According to Greenhow, Robelia, and Hughes (2009), the largest group of Internet users in the USA is school-age youth who go online several times per day, spending an average of nine hours per week on networking. This engagement in social media (and the use of other technologies in school as well) has had a positive influence on students’ performance and has increased their preparation for the job market. This connection between the younger generation and technology may indicate that the traditional methods of teaching are no longer effective. However, research has shown that there are negative effects of online learning as well, such as a lack of interaction, isolation, a lack of motivation, and poor communication (Waha and Davis, 2014). Therefore, a mix of both the traditional and technology-based approaches has been suggested as an optimal method for learning.
Blended learning incorporates the advantages of both methods to fit students’ needs and wants, as students prefer the blended type of learning because of its flexibility in scheduling, the diversity of tasks in class, the interaction with classmates and instructors, the quality of learning, and so on (Waha and Davis, 2014). According to O'Connor et al. (2011), the combination of both online and traditional learning approaches provides flexibility in terms of times and locations, offers both online and physical lessons, and provides the positive sides of traditional and technology-based methods. Renner, Laumer, and Weitzel (2016) emphasize that the appropriateness of blended learning differs based on the learning content, the different phases of technology use, and the skillset of the students. This is a further indicator that a change to blended learning environments needs a change process and preparation on both the students’ and instructors’ sides.
Many researchers have investigated the effects of introducing social media and other technologies to classrooms. Some of them have argued that the implementation of various media in lectures has an unlimited number of possibilities (Garrison and Kanuka, 2004), so there is a need to identify the technologies that will best help students to learn the material, rather than to engage them in all possible options in the hope that one works.
Garrison and Kanuka (2004, p. 97) identified an important feature of blended learning that they called “the ability to facilitate a community of inquiry,” which balances the large amount of information on the Internet with open communication between students and the instructor. The social element in the sense of bringing people together and the physical presence of the instructor are central to blended learning courses. Attempts to implement blended learning should focus primarily on designing a new learning and teaching experience to increase efficiency in higher education, rather than including all types of technologies to deliver the same content (Dang, Zhang, and Amer, 2019; Garrison and Kanuka, 2004). Therefore, the implementation of blended learning is usually accompanied by a redesign of the conceptualization of a course. The importance of interaction between instructors, peer teaching assistants, and students based on a social network for a blended class’s success is further analyzed in Dang et al. (2019). Communication structures are therefore considered an important design element and success factor of blended learning.
Related Work
When the technology-based delivery method of learning emerged as a new method of learning in higher education, much of the research was dedicated to the psychological aspects of learning with this approach (Aristovnik et al., 2017). However, as Aristovnik et al. (2017) argued, there is a gap in the research in terms of evaluating students’ perceptions of and attitudes toward blended learning courses. There is also a need to evaluate the difference, if any, in students’ performance between learning with a blended approach and learning in traditional face-to face environments or on online courses.
Researchers have studied students’ attitudes toward technology-based learning, but only a few studies have compared students’ performance in traditional classes with that in technology-based learning environments. Gupta and Bostrom (2009), Eryilmaz, Ryan, van der Pol, Kaemvilas, and Mary (2013), Cheng (2014), and Sun, Fang, and Zou (2016) can be named as examples for such studies in the IS context.
Although most studies have found a positive influence of technology-based learning on student performance, some have found student dissatisfaction with the approach. Harding, Kaczynski, and Wood (2012) found that students placed the highest value on the flexibility of class times and locations, as they could work when they felt productive while also benefiting from the weekly contact sessions, since they believed that face-to-face interaction was an important aspect of learning. Students also found the cooperative learning element favorable when they were engaged in group projects and could learn from each other. Other findings from this study indicated that students in blended learning environments became more independent and organized, since this approach to learning requires more self-discipline and responsibility than traditional face-to-face lectures do. Many other studies have also shown students’ positive attitudes toward blended learning courses because of their good course structure, increased motivation, the communication medium used, the instructor’s characteristics (timeliness, attitude toward the technology used, enthusiasm), the facilitating conditions (organizational and technical support when
using a specific technology), and the ease of use of the technology in class (Dang et al., 2016; Jung, Choi, Lim, and Leem, 2002; So and Brush, 2008).
However, some studies have indicated that students do not always perceive blended learning as positive. Parkinson et al. (2003), professors at a public university in the USA, compared students’ satisfaction and preferences for various delivery methods used in four courses and found that the students who were the most satisfied with the content, lectures, and the process of learning were those in courses taught using the traditional approach to learning. The least satisfied students were those taught on campus with the blended learning approach, primarily because of challenges they faced with the technology. Allen et al.’s (2002) empirical study found that students were more satisfied with traditional methods of learning than they were with distance learning formats that included multiple technologies. However, these negative results came from studies conducted in 2002 and 2003, and the technological challenges that existed then have since been solved because of the increased maturity of education technology.
As Allen et al. (2002) suggested, measuring students’ satisfaction and attitude toward learning methods is not the only way to evaluate the various approaches, as differences in performance and students’ learning outcomes are also important. This kind of analysis can be challenging to conduct, as the same cohort of students must take both the traditional course and the blended learning course. The approach we outline in this paper allows us to analyze not only students’ perceptions about blended learning courses but also the performance differences between students’ performance in a technology-based learning environment and a traditional face-to-face environment using students’ grade point averages (GPAs).
Research Method
A mixed-method approach using qualitative and quantitative methods was used to determine the impact of blended learning on a student group. Many studies have analyzed either students’ perceptions and attitudes or their performance, expressed in the quantitative results of exams or similar testing procedures (Renner, Laumer, and Weitzel, 2014). However, those that focused on students’ motivations and appreciation based on their individual preferences and exposure to peer groups did not consider that a positive attitude toward technology use does not automatically lead to better results on a course. As for the studies that only looked at performance using the results of a given setting, recommendations for improvements or variations in the technology used cannot be derived from this kind of research, as only the results of the complex dynamics of exposure to blended learning are presented in the quantitative test results.
Our analysis includes an analysis of perceptions and attitudes while comparing satisfaction with the setup and performance on the course to the extant results from courses conducted in traditional settings. A detailed overview of the methods used is presented in Table 1.
Table 1. Mixed-Method Approach
Conducted Within Group Conducted Between Groups Qualitative Quantitative Focus Group: Perception, Motivation Survey: Motivation, Attitudes Comparison of Exam I and II: Performance Comparison of Course Evaluations: Satisfaction Comparison of Exam II with a consideration of GPAs: Performance
Table 1. Mixed-method approach to analyzing the effects of blended learning.
Two focus-group discussions were conducted to qualitatively evaluate students’ perceptions and attitudes toward technology-based learning. The focus-group participants also filled out a survey to collect standardized, quantitative feedback. Then, students’ grades in the final exam in 2018, their GPAs, and their course evaluations were compared with the corresponding results of students from the previous year. The students on the 2018 course took two different exams, and the better grade counted in evaluating the course. Exam I was designed around the content and setup of the 2018 blended-learning-based course, while Exam II was designed blended-learning-based on the structure and content of the 2017 course to focus
on formal knowledge derived from the textbook. Exam I consisted of open questions and related to a short case provided with the question sheet. The questions were focused on different steps of an enterprise system implementation project and related to the material that was provided as blended learning content. Exam II consisted of a mixture of single choice, multiple choice, and open questions, all of them focused on the textbook that was used in both the 2017 and 2018 courses. As Exam I did not contain choice-based questions or any answer support (such as by providing categories, etc.), a learning effect from Exam I for increasing performance in Exam II was not possible. Furthermore, the questions for the two exams were carefully selected to avoid having the same question or topic area appear in both exams.
This strategy was used to assess the impact of technology-based learning on students’ performance and learning outcomes, which might differ from their perceptions and attitudes toward technology-based learning. One important aspect of our research design is the measurement of students’ performance in reality and not in an artificial setting, as researchers are often uncertain about the motivations of subjects in scientific investigations. The setup used in this study was based on students’ intrinsic motivations, as their behavior in the exam (i.e., the quantitative result of their performance) was the immediate foundation of the grade they received for the course.
The study site was the Enterprise Systems module at the University of Liechtenstein, which used the technology-based learning approach. The second-year course from the undergraduate program in business administration uses content and exercises related to models and theories in the field of enterprise systems. It focuses on understanding and developing new strategies for the successful implementation of enterprise systems using a case study. The Enterprise Systems module was taught with the traditional face-to-face approach to learning using PowerPoint presentations and Moodle as a learning platform for distributing handouts and examinations. However, the course structure and organization changed in the 2018 spring semester to implement a blended learning setup.
Four online and offline instructional approaches were adopted in the course. First, the lectures employed a textbook covering the main concepts and theory on the topic of enterprise systems. Second, online videos were presented to students as additional material on several topics from experts in the field. Third, an extended case study was used that required students to work in groups and present their findings in class in the form of a board presentation. Fourth, to allow students to work independently outside of class at their own pace and at times that suited them, presentation sessions were changed to unguided group-work sessions in which students were required to read chapters from the book, watch several online videos, and work on specific deliverables. The instructor also prepared a series of video lessons that systematically walked students through all of the material. Because of the university’s regulations, neither attendance nor performance during the lectures could be graded, as the course grade was derived from a ninety-minute final exam.
Two focus-group discussions were conducted with sixteen students in the Enterprise Systems module to explore their attitudes toward and actions/reactions to the technology-based learning approach used in this course. Focus groups as a qualitative method of data collection are often used as a single method for analyzing group norms, as part of a multi-method design to collect information that can be used in later stages of research, to extend data collection carried out with other methods, to support or contradict the results of the extant research, and so on (Krueger and Case, 2009). Focus groups are also often used to collect data prior to using another quantitative research method to identify hypotheses or theories to test in later stages of the study (Calder, 1977). Considering that this was the preliminary phase of this project, it was appropriate to use the focus groups to identify and explain some aspects of the technology-based learning approach.
The focus groups organized for the purpose of this study were used to collect students’ opinions and judgments about learning on a blended learning course. The focus groups were conducted on May 28, 2018, between the end of the lecture period and before the semester’s final exam. The topics that were discussed during the focus-group sessions were based on a discussion guide that was developed in coordination with the instructor and the teaching assistant of the Enterprise Systems module. To avoid bias, the focus groups were moderated by external researchers (two per group), and neither the lecturer nor the teaching assistant were present. The participants in this qualitative research study were sixteen students on the Enterprise Systems course, all of whom were between 20 and 29 years old. One focus group was conducted in English and one in German, as some students spoke German, and some spoke
English. The focus-group sessions lasted around an hour each and took place in the University of Liechtenstein’s facilities.
The focus groups began with the facilitators outlining the purpose and format of the discussion and setting the groups at ease. They told the participants that the discussion was informal, that everyone was expected to participate, and that divergent views were welcome. The focus-group discussions were recorded using both tape-recording equipment and the moderators’ extensive hand-written notes, which reflected the content of the discussion and any salient observations of nonverbal behavior such as facial expressions, hand movements, and group dynamics.
A focus-group discussion should be used as a preliminary method in collecting data, and its results should not be presented as quantitative research (Calder, 1977). Therefore, we conducted a second analysis of the students’ exam performance, changes in their GPAs, and student evaluations to access the impact of technology-based learning on student performance.
Five hypotheses were developed to guide the analysis:
H1a: Students who are taught with technology-based learning perform better on average than those taught with a traditional face-to-face learning approach do.
H1b: Students perform better on the case-based exam than they do on the traditional exam.
H2a: Students from the technology-based class evaluate the Enterprise Systems course more positively than students from the traditional class do.
H2b: Students from the technology-based class evaluate the instructor of the Enterprise Systems course more positively than students from the traditional class do.
H3: The technology-based learning approach improves the performance of all students, not only the best ones.
The data on the students’ performance were gathered to test H1a and H1b. We compared the 2017 exam results when students were taught using a traditional approach with the 2018 exam results when students were taught with a case-based learning approach. Students who were taught with the case-based approach had the opportunity to take both exams—the classical (as provided to the course participants in 2017, called “Exam II” in Table 1) and the case-based exam (in close relation to the content of the blended learning class of 2018, called “Exam I” in Table 1)—on the same day, with the best grade counting as the final grade. This analysis of real-life data was used to compare which exam the students performed better in on average.
Hypotheses H2a and H2b were tested by comparing the 2017 students’ evaluations of the lectures and instructor with those of the 2018 students.
To determine whether the technology-based learning approach improved all students’ performance (H3), we compared the exam results of the 2018 students with the total GPA they had achieved to date during their studies in the program. This allowed us to relate the individual performance in the exam to the average performance of each student and analyze deviations.
Results
Main Findings of the Focus-Group Interviews
The focus-group participants were very open and shared their impressions and opinions in an open manner. Table 2 gives a structured overview on the discussion’s results. The following section contains a more in-depth description of the results.
Table 2. Focus-Group Discussions
Themes Results of the Focus-Group Discussions Students’ perceptions on learning process • Including cases and online videos in a
lecture would be effective if there was motivation to attend the class and
participate in those activities.
• There was a preference for a combination of traditional and technology-based learning; although students were open toward the idea of including cases and other blended learning methods.
• Learning through cases and online videos, as in the Enterprise Systems lectures, was completely new to students and they preferred the course that was structured as a combination of
traditional and modern types of learning.
• The lack of opportunity to ask questions and the lack of interaction in class during the online videos was a disadvantage that meant students preferred traditional learning more.
• With a blended leaning approach, it was more difficult to predict the possible exam questions.
Students’ perceptions on learning outcomes • Performance was not affected by the technology-based learning approach due to students lacking the motivation to attend lectures.
• The technology-based method of learning was too fast a movement away from an entirely classical system that most of the students were used to.
• The technology-based learning approach would be really beneficial in the near future, once students had got used to such methods.
• If there was the motivation to go through all the provided materials, the students would have shown better performances in the exam, and this method of learning would have had a major positive impact on them.
Advantages and disadvantages of the blended
learning approach • The flexibility in terms of time schedules and the possibility of going through the online videos many times after the lectures were some of the advantages.
• Students did not miss much when they could not attend the lectures since they could find most of the materials online.
• Students were provided with a wide variety of materials and were not limited to the book or the slides.
• Some of the disadvantages included difficulties in understanding the content of the online videos, a lack of interaction and few opportunities to ask questions,
the high level of effort involved in covering the material, and an increased possibility of getting distracted
compared to the traditional learning approach.
The role of the instructor in blended learning
classrooms • Considering the effort involved in providing a wide variety of materials and the commitment to making videos that walked the students through all the steps, the instructor’s role was to facilitate the process of learning.
• The instructor’s knowledge on the topic and his/her enthusiasm to pass on that knowledge were impressive, yet the students felt they could benefit more from these aspects through classical lectures.
Table 2. Overview of the Focus-Group Discussion Results.
The focus-group discussion opened with students evaluating the process of learning the material from the Enterprise Systems lectures. Students thought that including cases and online videos in lectures would be effective if they were motivated to attend the classes and participate in those activities. However, with no mandatory attendance at the lectures and no grading of the cases that were prepared, most of the students were not motivated to participate in class or to watch the online videos they were provided with. The students were open to including cases and other blended learning methods in the lectures, but they preferred a combination of traditional and technology-based learning.
Most of the students thought that the cases should be part of the exercises, rather than the lectures, and that the lectures should stick to traditional learning approaches with slides and the instructor teaching the theory. Their view was that including certain cases and online videos in the exercises would have been more effective and they would have been more motivated if attendance and participation in the classes were mandatory and they received credit for their contributions. In addition, students would have liked the chance to practice everything they had learned regarding theory during the lectures.
Most of the students claimed that the online videos from open.sap.com were difficult to understand and that the lack of opportunities to ask questions and interact in class during the online videos was a disadvantage that made them prefer the traditional forms of learning. In general, the students believed that, in using the case-based learning method, they captured the material quickly and without much effort. Most also thought that with the material covered during the lecture, which included cases and content from open.sap.com, it was difficult to predict the exam questions, and that the exam questions based on a certain case would have no right or wrong answers, which made it difficult to study for.
Students were of the opinion that their performance was not affected by the technology-based learning approach, as when there was no motivation to attend the lectures and cover the necessary materials, there was no improvement in performance.
The participants argued that the technology-based method of learning used in the Enterprise Systems lectures moved too fast compared to the traditional system that most of the students were used to. They thought that the technology-based learning approach should be introduced to students gradually to have an effect on their performance, initially in combination with traditional lectures. In addition, some of the participants from the English-speaking group thought that the technology-based learning approach would be highly beneficial in the near future, once students had become accustomed to such methods.
The participants were asked whether they thought they would perform better in the exam with the blended leaning method than they would perform with the traditional approach to learning. Some thought that if they were motivated to go through all the materials provided, then they would perform better in the exam, and that this method of learning would have a major positive impact on them. Since attendance at the lectures was low, students were asked whether they would participate in traditional lectures even if it
was not mandatory, and most in one group answered that they would, while those in the other group had no clear preference.
Participants were asked about the advantages and disadvantages of technology-based learning, considering the advancements in technology and its effect in our society. Participants listed the flexibility in terms of time schedules, the ability to go through the online videos many times after the lectures, the ability to combine work and study, and the wide variety of materials as some of the benefits of this method of learning.
Participants from both focus groups mentioned that the disadvantages of technology-based learning that they encountered during the Enterprise Systems course included difficulties in understanding the content of the online videos, the lack of interaction and ability to ask questions, the effort and self-directed work required to cover the material, and the increased likelihood of being distracted.
Some of the participants contended that, after providing a wide variety of materials, including making videos that walked students through all the steps, the instructor’s role was to facilitate the process of learning. They cited the instructor’s knowledge on the topic and his enthusiasm for passing it along as impressive but thought they could benefit more from these aspects if they received traditional lectures from the instructor. Although the communication with the instructor was effective, most participants saw a need for additional feedback from his side and discussions with other groups in the class.
The students indicated a preference for online videos over professional guest lectures as they could access online lectures at any time, and for presentations, where students present their work in front of an audience and receive feedback on it. However, students thought that the traditional approach to learning should not be eliminated, as a combination of face-to-face lectures, online videos, presentations, real-world cases, and guest lectures would be much more effective from their perspective. Some students felt it was important to maintain the interaction with the instructor and their classmates.
After the focus-group discussions, the participants were given a set of statements regarding the learning process in the Enterprise Systems lectures and were asked to evaluate each statement on a scale ranging from 1 to 5, with 1 being “strongly disagree” and 5 being “strongly agree.”
Respondents ranked most of the questions with a neutral score close to 3, but they agreed to a high degree with the statements “More effort was required in the Enterprise Systems course compared to other courses” (average rating of 3.4) and “I want more technologies to be included in future lectures” (average rating of 3.3). Participants’ average rating of the statement about the instructor’s availability for feedback and assistance was only 2.0, which complies with the result from the discussion where students stated that they wanted more feedback and input from the instructor.
Data Analysis and Empirical Results
A key methodology used in this paper was an experimental design that could be evaluated using a t-test for two independent samples with similar variances. The effect of the blended learning setup was tested by offering two different exams to the group: Exam I, which related strongly to the methods implemented in the 2018 blended learning setup, and Exam II, which related purely to the content of the textbook and resembled the 2017 course’s execution. This methodology allowed us to make two types of comparisons: First, we compared students’ performance in the final exam in 2018 when the experiment was implemented with the students’ performance in the final exam in 2017 when the course was taught with a traditional learning approach. Second, as all students in the 2018 cohort participated in the experiment, we offered them the opportunity to take both exams and compared their performances in these exams. By comparing the students’ performance in the final exam in 2018 with the students’ performance in the final exam in 2017, we tested H1a: Students who are taught with technology-based learning perform better on average than those taught with a traditional face-to-face learning approach do. The t-test conducted to analyze the difference between the students’ performances does not provide support for H1a. At the 0.05 level of significance, students taught with technology-based learning did not perform better than those taught in a traditional face-to-face learning environment. The result had a p value of 0.9999, which is considerably greater than 0.05, indicating that the results are not significant.
Comparing the results of the two exams taken by the 2018 students tested H1b: Students perform better on the case-based exam than they do on the traditional exam. The paired t-test does not provide support for this hypothesis. At the 0.05 level of significance, students did not perform better in the case-based exam than they did in the traditional-approach exam. The result had a p value of 0.9065, which is considerably greater than 0.05, indicating that the results are not significant.
To assess the impact of technology-based learning on students’ performance and learning outcomes, we performed a comparison of student evaluations of the course’s lectures and the instructor from 2017 and 2018. These evaluations are performed every semester and are managed by the University of Liechtenstein after students have completed a course. The evaluations contain sentences that students rank on a scale ranging from 1 to 6, where 1 is “strongly disagree” and 6 is “strongly agree.”
Through this comparison, we tested H2a: Students from the technology-based class evaluate the Enterprise Systems course more positively than students from the traditional class do. The results show that, on average, students from 2017 ranked each of the sentences higher than did students from 2018, lending no support to H2a.
We used the same comparison to test H2b: Students from the technology-based class evaluate the instructor of the Enterprise Systems course more positively than students from the traditional class do. The results of the comparison showed that, on average, there was not a significant difference between the rankings of students in 2017 and 2018 in terms of their evaluations of the instructor, so H2b was not supported.
The evaluation of the lectures and the instructor was in keeping with the evaluation that the students gave during the focus-group discussions, as their evaluations of the efficiency of technology-based learning and their performances were mostly neutral.
The data collected on students’ performance were also used to determine whether the technology-based learning approach improved all students’ performance, not only that of the best students. Based on the data gathered on students’ performance in the exam and students’ GPAs, we specified a simple linear regression model to determine whether participants’ performance in the Enterprise Systems exam correlated with their overall GPAs and to test H3: The technology-based learning approach improves the performance of all students, not only the best ones.
The results from analyzing the data in the statistical software package R showed that there is a positive relationship between students’ exam scores and their overall GPAs such that the higher a student’s GPA is, the higher the score he or she receives in the exam from a blended learning course. However, the p
value is 0.05139, which is greater than 0.05, so it is not significant. The positive relationship between students’ exam scores and their GPAs could be the result of the small sample size, which is also one of the limitations of this study, as if the sample size is too small, the alternative is very difficult to prove.
Discussion
This paper reports on the outcome of a mixed-method-based analysis from teaching data and two focus-group discussions about the use of a blended learning approach on the University of Liechtenstein’s Enterprise Systems course. The study results do not indicate a strong impact of technology-based learning on students’ performance, and students’ performance is not affected by the introduction of technology-based learning, regardless of their overall GPAs and their performance on other courses. It is possible that these results do not provide strong evidence because the sample size was small. However, the findings do indicate that increased technology use in a learning setting does not always lead to performance improvements but has ambiguous effects. Thus, the ambiguous influence of technology use that has been reported in earlier studies is supported in our analysis.
Consistent with the existing literature, our study did not find a positive effect of technology-based learning in terms of students’ evaluations of the course and the instructor, since students from a technology-based learning class evaluated their course more negatively than students from a traditional learning class did. In addition, no significant difference in evaluations of the instructor between students from technology-based and traditional classes was found.
The results of the data analysis align with the findings derived from the focus-group discussions with students of the technology-based Enterprise Systems course. These results indicate that, although many benefits have been associated with the introduction of technology-based learning such as flexibility in the time schedules, the ability to work and attend school at the same time, the wide variety of materials offered, and the ability to access the materials at any time, this learning approach did not affect students’ performance.
Conforming with the findings of the focus-group discussions, the data analyzed from the two independent samples—exam scores of students in 2017, who were taught with the traditional approach, versus the exam scores of students in 2018, who were taught with the technology-based learning approach—provide no statistical evidence that students who are taught with technology-based learning perform better than those taught with the traditional face-to-face learning approach do. The main reason for this result was the lack of motivation for students to attend the lectures. The students themselves stated that if they had been motivated to go through all of the materials provided, they would have performed better in the exam, and this method of learning would have had a major positive impact on them. However, because attendance at the lectures was not mandatory and the case presentations were not graded, most of the students lost the motivation to participate in class. We could not manage this issue, since the university’s policy does not allow for mandatory attendance at lectures. However, this result shows that when a technology-based learning method is introduced, students must be motivated to go through all of the material provided using the technology that delivers it. Following this recommendation would require redesigning the university’s regulations to allow for more flexibility in grading students’ performance during the semester. The findings from analyzing the data and the focus-group discussions show that when there is no motivation to attend the lectures or cover the necessary materials, then there is no improvement in performance, since students prioritize other courses’ tasks that have clear deadlines during the semester.
As for students’ attitudes toward blended learning—the combination of case-based didactics and technology-based learning—, the students in our study were open to the idea of including cases and other blended learning methods in the lectures, but they preferred a stronger focus on traditional lectures when combining traditional and technology-based learning. The findings comply with the literature review and other studies that have shown that students value interaction with the instructor and other classmates, which they usually have in the traditional learning approach. In the focus-group discussions and in informal talks, the students showed great interest in the changed setting of the course. They understood this new setup as a challenge and as a possibility to experience new modes of learning that could be highly beneficial for them. The initial attitude was therefore very positive and open toward the changed course design.
However, during the semester, their focus shifted. Other courses were considered of higher priority as the semester schedule forced them to submit deliverables in given time frames during the semester. The degree of freedom provided in the blended learning class finally allowed for distraction. The students’ opinions changed and in the end uncertainty and stress were the dominant feelings when the final exam was coming closer. The dynamics during the semester would therefore need more consideration than we could manage in our case. The focus-group discussion results provide many insights that serve as the basis for improvement suggestions regarding these dynamics. Our recommendations for measures for a successful change to blended learning to be taken into consideration on the individual, class, and university levels are provided in the Study Implications section.
The students did not welcome the introduction of technology-based learning and the inclusion of cases in the course because of difficulties in understanding the content through online videos, the lack of interaction and the chance to ask questions, the great effort required to cover the material by themselves, and so on. However, the students agreed that the technology-based learning approach would be beneficial once the students had become accustomed to such methods, which should be in the near future. The students agreed that the idea of a combination of face-to-face lectures, online videos, student presentations, real-world cases, and guest lectures was promising and that it could be highly effective. Our results fit with earlier findings and go beyond them. Other studies, as shown by Renner et al. (2014), showed ambiguous changes in performance and perception when switching to blended learning environments as well. In an early study, Kendall, Kendall, Baskerville, and Barns (1996) analyzed both performance and perception when switching to interactive, computer-based case studies in teaching
systems analysis and design classes. However, their analysis only covered quantitative data and did not allow further conclusions to be drawn regarding the attitude or motivational drivers that underlie the observed results.
Our research design, which enriched the quantitative data with qualitative data gained from the focus groups, allowed us to go a step further and analyze the reasons why students’ performance and perception did not improve in a blended learning environment. Based on these insights, recommendations for a successful implementation of blended learning were derived, which are presented in the Study Implications section. This in-depth analysis can serve as a starting point for further research while being a guiding line for change processes toward blended learning in universities.
Conclusion
This study investigates the effect of technology-based learning on higher education students’ performance. The study analyzes students’ perceptions of and attitudes toward blended learning courses based on their experience on an Enterprise Systems course. The research empirically tests the difference between students’ performance when they are taught with technology-based learning versus when they are taught with the traditional approach to learning. As the sample size used in this paper was small, it was difficult to show differences in the students’ performance, but the results did not show a negative impact from a blended learning approach so much as ambiguity related to the use of technology in teaching. On the contrary, students were open to the idea of technologies and blended learning methods in the classroom, as long as there was still interaction with the instructor and other classmates.
The results of the data analysis and focus-group discussions indicate that for the blended learning approach to affect students’ performance, students must be motivated to attend the lectures and to cover all of the materials provided to them, as they prioritized other courses’ deliverables that were mandatory and had deadlines over the Enterprise Systems course’s voluntary submissions. Therefore, contemporary blended learning approaches do not by default increase the level of intrinsic motivation, although such a result was expected because of Harding and colleagues’ (2012) results. Motivation must be explicitly created in a blended learning environment to ensure learning success and cannot be expected to be a side effect of implementing blended learning.
The environmental conditions must also be considered when blended learning is implemented. Our results show that the flexibility that is inherent in an open learning space is used to prioritize other courses where participation during the semester is monitored more closely. That is, when blended learning is introduced, universities must understand students’ workloads and what drives their attention allocation. Providing online sources and giving flexibility in working through the course content can lead to a situation in which students are under the illusion that they can work with the course content at any time and do not have to work regularly. This situation can lead to frustration and stress during exam preparation when students cannot even contact the lecturer to ask questions. Both changes in the ecosystem’s regulations and definitions of other courses’ requirements should be considered to create a supportive working environment for implementing blended learning.
Study Implications
This paper addresses various aspects of applying technologies in classrooms, students’ attitudes toward the blended learning approach, and its impact on higher education students’ performance. Our findings suggest that success stories and the resulting unrestricted plea to use technology to establish blended learning may be misleading.
This paper contributes to increasing awareness of the challenges in providing blended learning and getting students on board. The study’s results should be considered when universities implement technology-based learning approaches. Several recommendations for the implementation of blended learning environments can be derived from our analysis and discussion. These can be structured along different granularity levels as follows:
Individual Level
- Make sure that students understand the setup of the blended learning environment and all didactical components (e.g. online lectures, quizzes, cases, their own contributions such as presentations).
- Share your motivation for using blended learning and make them understand that they play an active role in this setup.
- Whenever possible, give them a choice between a traditional class and a blended learning class. This allows for the comparison of the performance in achieving the learning objectives and perceptions on the course between the two groups. Kendall et al. (1996) show how such a comparison can be used for further improvement.
Class Level
- Provide structure to the students by requesting their own contributions on a regular basis. This ensures that they will deal with the course content from the beginning. Alternatively, students could think that they can watch all the recorded material in a batch and prepare efficiently for a final exam this way. Our experience showed that this kind of procrastination finally creates stress and uncertainty among the students.
- Make sure that they stay in touch with the other students, peer teaching assistants, and with the instructors. Interaction between these groups is an integrative part of traditional learning settings. However, in blended learning environments the communication between these groups needs to be managed and supported by suitable measures. Dang et al (2019) developed a model to understand the dynamics and importance of social networks in blended learning environments.
University Level
- Consider the suitability of learning goals and learning objectives for blended learning classes. A suitable formulation can support the use of blended learning, for example, by focusing on a holistic understanding and transferring knowledge instead of replicating knowledge provided in a lecture or textbook.
- Give responsibility to lecturers and nominate some courses where different modes of teaching can be tested. The design of courses and the inclusion of blended learning elements must consider previous knowledge of the students and the suitability of the topic. Renner et al. (2016) analyzed the appropriateness of blended learning in different phases of a learning process while considering the previous skills of the group.
- Adapt regulations to allow for flexibility in grading and examination setup as this will enable rewarding active participation and contributions during the lecture period. In our case, grades were only based on a final exam. Results showed that this was a major drawback for motivating students to actively participate in the board presentation sessions.
- Curriculum committees, the rectorate, deans, and other representatives of the university should be supportive on this path toward a blended learning environment. It must be clear that testing and comparisons between different setups are necessary to improve the performance of blended learning. This is especially important as the lead time of such a project is one or more years. Usually, courses are offered once a semester or even just once a year. To enable a comparison between traditional and blended learning environments and to run a second execution of a course, several semesters are necessary. Furthermore, structures for gaining reputation by implementing new modes of teaching (e.g. an award for innovative teaching) and structures for knowledge-sharing need to be established.
Change Process
Regarding the change process, the technology-based learning approach should be introduced to students gradually, first in combination with the traditional approach, as a step-by-step approach to blended learning.
Based on the results from our analysis of students’ perceptions, rapid movement from a traditional system that students are accustomed to into a purely technology-based learning environment might fail because of students’ inability to manage the combination of technologies offered to them. Our results also indicate
that when blended learning is implemented in classrooms, students must be motivated to participate by means of such measures as mandatory attendance at lectures and credit for work during the semester. Another major contribution of this paper is in the design of a mixed-method approach to evaluate both the personal, motivational aspects for participation in blended learning settings and the measurable performance change that is due to a blended learning approach while at the same time compensating for possible year-on-year fluctuations using the GPA as a basis for comparison.
The present study does not address several aspects of the topic, providing the opportunity for further investigations. For instance, this study investigates the impact of blended learning using a course with only sixteen students. The evaluation method developed in this paper would benefit from further testing with larger samples.
In addition, any study on technology-based learning can test only a few combinations of the vast number of possibilities for creating a technology-based course. This study provides evidence of and insights into a certain combination of technologies and other approaches in class, including online videos, case studies, unguided group-work sessions, video lessons from the instructor, and presentation slides. Future studies could focus on the effect of other types of technologies on students’ performance and their attitudes toward this method of learning.
References
Allen, M., Bourhis, J., Burrell, N., and Mabry, E. 2002. “Comparing student satisfaction with distance education to traditional classrooms in higher education: A meta-analysis,” American Journal of Distance Education, (16:2), pp.83–97.
Aristovnik, A., Tomazevic, N., Kerzic, D., and Umek, L. 2017. “The impact of demographic factors on selected aspects of e-learning in higher education,” International Journal of Information and Learning Technology, (34:2), pp.114–121.
Calder, B. J. 1977. “Focus groups and the nature of qualitative marketing research,” Journal of Marketing Research, (14:3), pp. 353–364.
Cheng, Y-M. 2014. “Extending the expectation–confirmation model with quality and flow to explore nurses’ continued blended e-learning intention,” Information Technology & People, (27:3), pp. 230– 258.
Chiang, F. K., Wuttke, H. D., Knauf, R., Sun, C. S., and Tso, T. C. 2009. “Students' attitudes toward using innovative information technology for learning based on theory of planned behavior,” International Journal of Advanced Corporate Learning (iJAC), (2:4), pp.9–14.
Dang, M. Y., Zhang, G. Y., and Amer, B. 2019. “Social networks among students, peer TAs, and instructors and their impacts on student learning in the blended environment: A model development and testing,” Communications of the Association for Information Systems, 44, pp. 764–782.
Dang, Y. M., Zhang, Y. G., Ravindran, S., and Osmonbekov, T. 2016. “Examining student satisfaction and gender differences in technology-supported, blended learning,” Journal of Information Systems Education, (27:2), pp. 119–130.
Donnelly, R. 2010. “Harmonizing technology with interaction in blended problem-based learning,” Computers & Education, (54:2), pp. 350–359.
Eryilmaz, E., Ryan, T., van der Pol, J., Kasemvilas, S., and Mary, J. 2013. “Fostering quality and flow of online learning conversations by artifact-centered discourse systems,” Journal of the Association for Information Systems, (14:1), pp. 22–48.
Eurostat. 2019. “Internet access and use statistics: Households and individuals,”
(https://ec.europa.eu/eurostat/statistics- explained/index.php?title=Archive:Internet_access_and_use_statistics_-_households_and_individuals; accessed September 5, 2019).
Garrison, D. R., and Kanuka, H. 2004. “Blended learning: Uncovering its transformative potential in higher education,” The Internet and Higher Education, (7:2), pp. 95–105.
Greenhow, C., Robelia, B., and Hughes, J. E. 2009. “Learning, teaching, and scholarship in a digital age. Web 2.0 and classroom research: What path should we take now?” Educational Researcher, (38:4), pp. 246–259.
Gupta, S., and Bostrom, R. P. 2009. “Technology-mediated learning: A comprehensive theoretical model,” Journal of the Association for Information Systems, (10:9), pp. 686–714.
Harding, A., Kaczynski, D., and Wood, L. 2012. “Evaluation of blended learning: Analysis of qualitative data,” in Proceedings of the Australian conference on science and mathematics education (formerly UniServe science conference). pp. 56–62.
Heinze, A., and Procter, C. T. 2004. “Reflections on the use of blended learning,” Salford University (http://www.usir.salford.ac.uk/id/eprint/1658/1/; accessed September 5, 2019).
Herold, B. 2017. “Analysis: 94 percent of school districts nationwide meet federal high-speed internet access targets,” Government Technology (http://www.govtech.com/network/Analysis-94-Percent-of-School-Districts-Nationwide-Meet-Federal-High-Speed-Internet-Access-Targets.html; accessed September 5, 2019).
Jung, I., Choi, S., Lim, C., and Leem, J. 2002. “Effects of different types of interaction on learning achievement, satisfaction and participation in web-based instruction,” Innovations in Education and Teaching International, (39:2), pp. 153–162.
Kendall, J. E., Kendall, K. E., Baskerville, R., and Barnes, R. 1996. “An empirical comparison of a hypertext-based systems analysis case with conventional cases and role playing,” Data Base for Advances in Information Systems, (27:1), pp. 58–77.
Kitzinger, J. 1994. “The methodology of focus groups: The importance of interaction between research participants,” Sociology of Health & Illness, (16:1), pp. 103–121.
Krueger, R. A., and Case, M. A. (2009). Focus groups: A practical guide for applied research (4th ed.). Thousand Oaks, CA: Sage.
Kwak, D. W., Menezes, F. M., and Sherwood, C. 2015. “Assessing the impact of blended learning on student performance,” Economic Record, (91:292), pp. 91–106.
Nguyen, D. D. 2011. “An empirical study of student attitudes toward acceptance of online instruction and distance learning,” Contemporary Issues in Education Research, (4:11), pp. 23–28.
O'Connor, C., Mortimer, D., and Bond, S. 2011. “Blended learning: Issues, benefits and challenges,” International Journal of Employment Studies, (19:2), pp. 62–82.
Ozkan, S., and Koseler, R. 2009. “Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation,” Computers & Education, (53:4), pp. 1285– 1296.
Parkinson, D., Greene, W., Kim, Y., and Marioni, J. 2003. “Emerging themes of student satisfaction in a traditional course and a blended distance course,” TechTrends, (47:4), pp. 22–28.
Renner, D., Laumer, S., and Weitzel, T. 2014. “Effectiveness and efficiency of blended learning: A literature review,” in Proceedings of the 20th Americas Conference on Information Systems, Savannah, GA, August 7-9.
Renner, D., Laumer, S., and Weitzel, T. 2016. “The appropriateness of blended learning across the phases of technology use,” in Proceedings of the 37th International Conference on Information Systems, Dublin, Ireland, December 11-14.
Schaber, P., Wilcox, K. J., Whiteside, A. L., Marsh, L., and Brooks, D. C. 2010. “Designing learning environments to foster affective learning: Comparison of classroom to blended learning,” International Journal for the Scholarship of Teaching and Learning, (4:2), pp. 1–18.
So, H. J., and Brush, T. A. 2008. “Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: Relationships and critical factors,” Computers & Education, 51(1), pp. 318–336.
Sun, H., Fang, Y., and Zou, H. 2016. “Choosing a fit technology: Understanding mindfulness in technology adoption and continuance,” Journal of the Association for Information Systems, (17:6), pp. 377–412. Waha, B., and Davis, K. 2014. “University students’ perspective on blended learning,” Journal of Higher
