Evaluation of E-Learning Implementation in Higher Education Using Multi-Criteria Technique

(1)

A Monthly Double-Blind Peer Reviewed Refereed Open Access International Journal

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 6

Evaluation of E-Learning Implementation in Higher Education Using Multi-Criteria Technique Husam Jasim Mohammed*1, Maznah Mat Kasim2, Izwan Nizal Mohd Shaharanee3

1,2,3

Department of Decision Sciences, School of Quantitative Sciences, University Utara Malaysia, 06010 Sintok, Kedah, Malaysia.

Abstract

E-learning plays an increasingly important role in developing educational growth, and it can play a significant role in preparing a new generation of teachers in higher education institutions. Universities in Malaysia are in the move of implementing information communication technologies in their teaching and learning activities. A study is required to evaluate potential e-learning towards the implementation of e-learning in universities in Malaysia. Therefore, the study discusses the evaluation of five e-learning approaches based on five identified criteria by using multi-criteria methods. A total of 95 participants consisted of administrative and academic staff, and post graduate students evaluated five e-learning approaches by using multi-criteria methods. They were asked to assess the relative importance of five e-learning evaluation criteria by using rank order centroid method. Furthermore, they also rated the performance of five identified e-learning approaches under each of the criteria. The overall performance of each e-learning approach was computed by using weighted product model method. The results suggested that flipped classroom is the most suitable e-learning approach, while ‘Strategic readiness for e-learning implementation’ found to be the most important criterion. The paper is suggesting a quantitative evaluation method for decision makers who are strategizing modern technologies in higher educational settings.

Keyword: E-learning, evaluating criteria, ROC, weight, WPM.

INTRODUCTION

(2)

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 7 was conducted in the Universiti Utara Malaysia (UUM) by sending the questionnaire to more than 700 people through emails, but only a total of 95 respondents answered the survey. The respondents consisted of 38 lecturers, 22 administrative staff, and 35 postgraduates students who evaluated the importance of the criteria and performances of the five e-learning approaches under each of the five criteria. Two Multi-criteria (MC) methods were used to analyze the relative importance of the criteria and to aggregate the overall performance of each e-learning approach. To achieve the objective of the paper, this article is organized as follows. The next section provides an overview of each e-learning approach.

E-LEARNINGAPPROACHESANDCRITERIA

he widely used strategies that rely on the use of modern technology to activate the digital learning strategies are: blended learning, ICT supported face to face learning, distance (remote) learning includes synchronous and asynchronous learning and flipped classroom. In this paper, five e-learning approaches (see Table 1) were evaluated and the best model was selected to be implemented in the selected public university. Rovai and Jordan (2004) defined blended learning as: “a mix of classroom and online learning that includes some of the conveniences of online courses without the complete loss of face-to-face contact” (p. 1). The blended learning is the mix of pedagogical approaches, using a variety of learning strategies with technology. It blends together processes of traditional learning and e-learning (Dos, 2014). ICT used as supporting with traditional classes to illustrate some of the ideas by the teacher with use ICT (Video, YouTube, pictures or PowerPoint) through data show in classroom.The flipped classroom has taken place in education as a modern teaching method (Osman, Jamaludin, & Mokhtar, 2014). It is a shift in the process from teacher-centred learning to student-centred learning (Bergmann & Sams, 2012), and it is a concept for active learning where students are provided with study materials like video lectures or online textbooks before they attend the class (Bishop, 2013). The e-learning is usually defined as a distance learning includes synchronous learning and asynchronous learning, and sometimes, it is also defined as a type of learning supported by ICT (N. Begičevid, Divjak, & Hunjak, 2007; Chao & Chen, 2009). The five e-learning approaches under study and the five evaluation criteria (Nina Begičevid, Divjak, & Hunjak, 2007) are as summarized in the following Table 1.

Table 1: E-Learning Approaches and Criteria

No. Criteria No. Alternatives

1 Human Resources. 1 Blended Learning

2 Specific ICT Infrastructure for

E-Learning. 2 Flipped Classroom

3 Basic ICT Infrastructure for

E-Learning. 3

ICT Supported Face-to-Face Learning

4 Strategic Readiness for

E-Learning Implementation. 4 Synchronous Learning

5 Legal and formal Readiness for

(3)

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 8

METHODOLOGY

The methodology consists of two main parts. The first part focused on the weights of e-learning criteria, while the second part was about the selection of suitable e-learning approach to be implemented in the selected university. The data were collected from a public university in Malaysia in 2016 through two sets of questionnaires which had been established by using Google Drive and then sent to participants through email. A total of 95 participants took part in the survey, and divided into three groups namely: administrative group (22 participants), academic staff group (38 participants), and postgraduate students group (35 participants). The first set is about the importance of criteria towards implementation of e-learning. Here, the Rank Order Centroid (ROC) was used as weighting method for the criteria where the respondents had to give points between one and five to each of the five identified criteria. The 95 evaluations were aggregated by using Geometric Mean (G. Mean) method. The second set of the questionnaire concerns about the rating of the performance of each of the e-learning approaches under every criterion. The scale of rating is 10 to 100, where the higher the rating means the higher the performance of the approach under the evaluation criteria. The G. Mean method was used once again to aggregate the 95 performances of each approach under each criterion. The Weighted Product Model (WPM) method was used to aggregate the weights of criteria and the performances of the e-learning approaches to determine the overall performance of the e-learning approaches.

RANKORDERCENTROIDMETHOD(ROC)

The ROC method was introduced by Barron (1992). It creates an estimate of the weights and minimizes the maximum error of every weight. Barron & Barrett (1996) found that weights obtained from this method were very stable. They showed that the expected value of the weight of each attribute could be computed using the following formula:

( ) ∑

( )

Where is the number of criteria, is the rank position of criterion , . This manner of getting the criteria weights is called ROC because these weights reflect the centre of mass (centroid) of a simplex defined by the ranking of the attributes. With more attributes, the error for ranked attributes will be much less (Roszkowska, 2013).

For example, the evaluation and the calculation of weights based on one of Decision-Maker (DM) are shown in Table 2.

Table 2: Calculation of the weight using ROC for one DM

Criteria Number

evaluation _Calculation ∑ Weights

1 ₍ ) 0.457

5 ₍ ) 0.040

4 ₍ ) 0.090

3 ₍ ) 0.157

(4)

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 9 WEIGHTEDPRODUCTMODEL(WPM)

The WPM is probably the most commonly used approach, especially in single dimensional problems. The criteria in WPM model as benefit criteria (compensation approach). Triantaphyllou (2000) explained that: “Each alternative is compared with the others by multiplying a number of ratios, one for each criterion. Each ratio is raised to the power equivalent to the relative weight of the corresponding criterion.

In general, in order to compare two alternatives and , the following product (Bridgman, 1922) has to be calculated:

( ) ∏ (

)

( )

Where is number of criteria, is the actual value of alternative in terms of the jth criterion, is

the weight of importance of criterion.

If the term ( ) is greater than or equal to one, then it indicates that the alternative is more desirable than the alternative (in the maximization case). The best alternative is the one that is better than or at least equal to all the other alternatives.

RESULTSANDDISCUSSIONS

The criteria weights were calculated by using ROC method. In this technique, the data collected from the respondents were analyzed and divided into three groups namely: administrative (G1), academic staff (G2), and postgraduate students (G3). Normalized weights of criteria were computed using Equation (1). The normalized weights of all groups are denoted as Ideal Weights (IW) for ROC method as shown in Table 3.

Table 3: The weights and rank for the criteria using roc method

Groups (Rank) (Rank) (Rank) (Rank) (Rank) G1 0.2379(2) 0.1129(5) 0.1144(4) 0.3325(1) 0.2023(3) G2 0.2684(1) 0.1855(3) 0.1612(5) 0.2038(2) 0.1810(4) G3 0.3268(2) 0.1018(4) 0.0805(5) 0.3327(1) 0.1582(3)

G. Mean 0.2753 0.1287 0.1141 0.2825 0.1796

IW 0.2809(2) 0.1313(4) 0.1164(5) 0.2882(1) 0.1832(3)

Based on the results in Table 3, it can be observed that the group 1 and 3 estimated (strategic readiness for e-learning implementation) to be chosen as the most importance than others, whereas group 2 preferred (human resources) as the most importance criterion than others. The IW based on G. Mean in ROC method presented that the is the preferred criterion than others, while , ‘human resources’ as the second most important criterion. The criterion , ‘legal and formal readiness for e-learning implementation’ is at the third ranking of importance, followed by , ‘Specific ICT infrastructure for e-learning’ and, , ‘basic ICT infrastructure for e-learning’. Correspondingly, the criteria weights using this method is showed in Figure 1.

(5)

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 10 This set of weights would be used in evaluating the e-learning approaches by using WPM method.

The following decision matrix in Table 4 displays the average of the judgments (G. Mean) for each alternative given by 95 participants for the five e-learning models. The weights of criteria are in brackets in row one.

Table 4: Decision Matrix of Criteria Weights and Average Evaluations of Each E-Learning Approach E-learning approaches C1 (0.2788) C2

(0.1525) C3 (0.1276)

C4 (0.2586)

C5 (0.1824)

Flipped Learning (A1) 83 83 80 84 85

Blended Learning (A2) 74 78 74 75 74

ICT & F-to-F Learning (A3) 65 62 62 60 60

Synchronous Learning (A4) 40 39 42 38 42

Asynchronous Learning (A5) 29 26 30 34 29

The result of the ranking of approaches is derived using Equation (2) are shown below.

( ) ( ⁄ ) ₍_{⁄ )} ₍_{⁄ )} ₍_{⁄ )} ₍_{⁄ )}

Similarly, we also get:

( )

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

Group 1 Group 2 Group 3 Ideal Weights

ROC weights Method

Criterion 1

Criterion 2

Criterion 3

Criterion 4

(6)

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 11

( )

Therefore, the best alternative is A1 (flipped classroom learning) since it is superior to all the other alternatives. Moreover, the ranking of these alternatives is as follows: Al >

A2 > A3 >A4>A5 (where the symbol (>) stands for “better than”). The flipped classroom learning is considered as the best maximization of expected benefits for e-learning implementation in Universiti Utara Malaysia (UUM).

CONCLUSIONS

This paper shows the utilization of multi-criteria methods in evaluating e-learning approaches under five identified criteria. The use of this type of quantitative method is very practical for evaluation purposes. Besides, the evaluation was carried out by those who were really involved whether directly or indirectly in the implementation of e-learning in a university. The results of the evaluation show that the strategic readiness for e-learning implementation found to be the most important basis of criterion from the perspective of the respondents from a public university in Malaysia. This finding has to be taken seriously since no matter how magnificent the technology is, the readiness still play the main role as operators as well as the users. Furthermore, the flipped classroom is the most preferred e-learning approach out of five approaches under study. The results of this study would give idea to the management of the university in their process of implementing modern technologies in the teaching and learning process.

REFERENCES

Al Musawi, A. S., & Abdelraheem, A. Y. (2004). E-learning at Sultan Qaboos university: Status and future.

British Journal of Educational Technology, 35(3), 363–367. http://doi.org/10.1111/j.0007-1013.2004.00394.x

Barron, F. H. (1992). Selecting a best multiattribute alternative with partial information about attribute weights. Acta Psychologica, 80(1), 91–103.

Barron, F. H., & Barrett, B. E. (1996). Decision quality using ranked attribute weights. Management Science, 42(11), 1515–1523.

Begičevid, N., Divjak, B., & Hunjak, T. (2007). Development of AHP based model for decision making on e-learning. Journal of Information and Organizational Sciences, 31(1), 13–24.

Begičevid, N., Divjak, B., & Hunjak, T. (2007). Prioritization of e-learning forms: A multicriteria methodology. Central European Journal of Operations Research, 15(4), 405–419. http://doi.org/10.1007/s10100-007-0039-6

(7)

International Journal in IT & Engineering

http://www.ijmr.net.in

email id- [email protected]

Page 12 Bishop, V. (2013). The Flipped Classroom : A Survey of the Research. Proccedings of the Annual

Conference of the American Society for Engineering Education, 6219. http://doi.org/10.1109/FIE.2013.6684807

Bridgman, P. W. (1922). Dimensional analysis. Yale University Press.

Chao, R.-J., & Chen, Y.-H. (2009). Evaluation of the criteria and effectiveness of distance e-learning with consistent fuzzy preference relations. Expert Systems with Applications, 36(7), 10657–10662. Dos, B. (2014). Developing and evaluating a blended learning course. Anthropologist, 17(1), 121–128. Hunjak, T., & Begičevid, N. (2006). Prioritisation of e-learning forms based on pair-wise comparisons.

Journal of Information and Organizational Sciences, 30(1), 47–61.

Khan, B. H. (2005). Managing e-learning: Design, delivery, implementation, and evaluation. IGI Global. Mourits, M. C. M., van Asseldonk, M. A. P. M., & Huirne, R. B. M. (2010). Multi Criteria Decision Making

to evaluate control strategies of contagious animal diseases. Preventive Veterinary Medicine (Vol. 96). Springer. http://doi.org/10.1016/j.prevetmed.2010.06.010

Osman, S. Z., Jamaludin, R., & Mokhtar, N. E. (2014). Flipped classroom and traditional classroom : lecturer and student perceptions between Two learning cultures , a case study at Malaysian polytechnic. International Education Research, 2(4), 16–25. http://doi.org/10.12735/ier.v2i4p16 Roszkowska, E. (2013). Rank ordering criteria weighting methods – a comparative overview 2.

Optimum.Studia Ekonomiczne Nr, 5(65), 14–33. http://doi.org/10.15290/ose.2013.05.65.02

Rovai, A. P., & Jordan, H. (2004). Blended learning and sense of community: A comparative analysis with traditional and fully online graduate courses. The International Review of Research in Open and Distributed Learning, 5(2).