Abstract- This paper is partially based on the Six Sigma process and its application to eLearning. Six Sigma process claims that focusing on reduction of variation and defects will improve a business product or a service. Its prestige and value is well-known within the manufacturing industry. In Asia-Pacific region it is beginning to be recognized within higher educational institutions too. The key component for consistent high quality can be achieved by application of Six Sigma techniques to the development and production processes of an eLearning courseware. The main Six Sigma aspects are explored in this paper, first by describing an “outside-in” approach, then by statistically analyzing usage of core learning objects (LOs) or components for quality in a courseware. The last section of this paper proposes adaptation of Capability Maturity Model (CMM) as a instructional technology solutions for an eLearning program.
Keywords- Capability Maturity Model (CMM), eLearning Courseware, Learning Objects, Quality, Six Sigma
I. INTRODUCTION
There is much evidence in ASEAN universities to characterize online learning as having come of age. It is a well known fact that in last five years using online technologies is more of an expectation than a novelty for today’s university students in ASEAN. In other words, there is abundant evidence that the Internet has become an integral part of university education in terms of accessing learning resources, communicating with faculty or classmates, and its overall usage [1, 2].
Likewise, eLearning courses are also becoming integral part of a university degree program at all levels- undergraduate, graduate and PhD programs. Furthermore, the installed base of infrastructure investment for online learning makes it clear that eLearning will become more and more relevant to students learing needs.
What is Six Sigma? Six Sigma is a management philosophy originally developed by the Motorola corporation located in U.S.A. It sets up extremely tough objectives for the collection of data and the analysis of results. Its main targets are to reduce defects in products and services. Six Sigma is a highly disciplined process that helps organizations to focus on developing and delivering near-perfect products and services. The word “Sigma” is a statistical term that measures how far a given process deviates from “perfection.” The central idea behind Six Sigma is that, if we can measure how many “defects” we have in a process, we can also work out how
to eliminate them and get as close to “zero defects” as possible [3].
The Six Sigma approach for a completed project includes five phases- Define, Measure, Analyze, Improve, and Control (DMAIC) for process improvement. For a new product and service development it includes slightly different phases labeled as Define, Measure, Analyze, Design, and Verify (DMADV). Six Sigma advocates a belief that: “Delighting the customers is a necessity. Because, if one entity don’t do it, someone else will!” A. Changing Business Processes in Higher Education
Globalization and instant access to information, products and services have changed the way customers conduct business. Old business models no longer work. Today’s competitive environment leaves no room for error. Hence universities must engage its students with high quality courseware and relentlessly look for new ways to exceed their expectations. Striving for quality in eLearning requires a college or a university to look at its business from the student’s perspective, not its own.
As early as 2002 S. Keith Hargrove and Legand Burge from University of Michigan College of Engineering, Ann Arbor, MI, USA proposed an approach adapted from industry that sought to assess, evaluate, and monitor variation in student’s performance in the curricula and recommended methods for improvement. By adapting the Six Sigma approach from industry to academia, their results provided a methodology to increase the retention rates of minority students to address the needs of highly skilled engineers needed in USA [3]. Since then Six Sigma has gained popularity because it has proved to be successful not only at improving quality of products but also essential students services provided in academia.
In another study conducted by Hakan Wiklund & Pia Sandvik Wiklund in Sweden Six Sigma was adapted as a company-wide approach for organizational improvement incorporating organizational learning. This study discussed various factors associated with manufacturing work organization and leadership that are essential for improving organizational learning and for stimulating the competency levels and motivation among personnel [4].
In manufacturing segment some companies have reported large savings after incorporating Six Sigma methodology. For example, Motorola, a leading member of a consortium of companies that developed the Six Sigma approach reported savings of over 11 billions
Applying Six Sigma Techniques to Improving the Quality of eLearning
Courseware Components- A Case Study
Kuldeep Nagi
1& Prof. Dr. Srisakdi Charmonman
2College of Internet Distance Education Assumption University
Bangkok, Thailand
dollars after introducing the Six Sigma techniques some 12 years ago. But eLearning and eTraining is a new and emerging market with huge potential. Historically many components of learning and training are rooted in traditional academic practices, not in business processes driven by technology.
II. METHODOLOGY
A. Applying Key Concepts of Six Sigma to eLearning Courseware Development Cycle
One of the key elements of Six Sigma is the use of measurement and analysis of data for process improvement. Why measure the usage of eLearning courseware learning objects (LOs) or components? It is evident that without data, we only have opinions. Measurement and analysis involves gathering data about products and processes. In other words, measurement and analysis of activities allow us to characterize, or gain a better understanding of key processes, products, resources and environments. Evaluation of results can help determine the status with respect to the plans. It can also help predict relationships among processes and the values we observe for key attributes that can be used to predict others. In the end, it can help improve the process and the product by identifying roadblocks, root causes, inefficiencies, and other factors. Capability Maturity Model (CMM) is integral to Six Sigma and can be customized to guide the process of improving quality of eLearning courseware components.
Since eLearning courseware is a product as well as service, application of Six Sigma can help in quality control of various learning objects (LOs) or components. This paper is a small effort in applying Six Sigma philosophy to eLearning courseware production processes.
Fig. 1. Learning Management System Menu- Reports
This paper is partly based the results of an ongoing work on the evaluation methods for eLearning platforms being used at the College of Internet Distance Education (CIDE), Assumption University of Thailand. The Six Sigma process adoption proposed here is based on different types of measurements available in the Learning management System (LMS) including active conference data captured in the logs. Fig. 1 illustrates the typical LMS user interface.
Six Sigma approach is adapted here for evaluating the quality based on usage of a learning object (LO) or a components. In this respect, one important aim of this preliminary work is to establish the use of Six Sigma evaluation methodology, and to gather data from LMS sources for a comparative analysis. Another purpose is to establish some benchmarks that can be measured to improve the quality of learning objects (LOs) or components. Quality assurance in eLearning programs is a growing concern among various stake holders, especially the universities in ASEAN.
A typical eLearning program in a college or a university adopts a one-sided view of the market, based on an ‘average’ performance of its courses offered in a academic program from its recent past. However, students do not judge a eLearning courseware on its average performance. Rather, they judge performance by its quality and service that they get. If applied carefully Six Sigma can reduce process variation in a courseware. It can also address improving the capability of those processes. For the purposes of this case study the Six Sigma is being applied to identifying-
i. Attributes of LOs or components that are most important to a student;
ii. The significance of Learning Management System (LMS) logs when it shows that the courseware learning objects (LOs) usage varies from what was expected?
iii. Ways of ensuring consistent, predictable processes to improve LOs or components the students’ use for enhancing their learning in a courseware?
B. Using Learning Management System (LMS) Reports- An “inside-out” approach for collecting and analyzing data
Six Sigma requires selection of appropriate measurement goals. Identifying quantifiable elements of goals is critical for quality assessment. In Six Sigma the investigators should identify indicators that will help address the questions and communicate the results of the analysis to others. It should also prioritize the indicators and identify the ones that will be most useful for refining the process. As mentioned earlier in this case study the researchers are exploring a very limited application of Six Sigma philosophy for improving eLearning courseware components. As a part of the academia, not the industry, the researchers had to look at the processes from the “inside-out.”
Every Learning Management System (LMS), whether proprietary or open-source collects huge amount of data about user’s activities.
Fig. 2 Six Sigma Steps
By understanding the transaction logs of students’ interactions in a LMS, the instructor or content expert can discover what students are seeing and using for enhancing their learning. Six Sigma approach can be customized to systematically evaluate the components of a eLearning given in the following steps. Fig. 2 given above illustrates the basic steps of the Six Sigma process. A brief description of each steps is given below [5]. This process is based on a very important hand book prepared by Software Engineering Institute at Carnegie Mellon University, Pittsburgh, USA in 1996.
1. Measuring Goals (G): Define what to be measured as Goals?
(G1) In this case study the goal is to explore the usability of learning objects (LOs) or components provided in a courseware?
2. Questions (Q): What questions should be asked or raised or how they could be answered?
In this case study following questions are being asked?
Q1. What is the frequency of usage various learning objects (LOs)or components in a courseware? Q2. What is the % usage of each learning object (LO)?
Q3. Which learning objects (LOs) or components of a courseware are under used?
Q4. What to do about learning objects (LOs) or components that have a very low usage or not used at all?
3. Indicators (I): What are the indicators of the usage of Learning objects (LOs) or components? The indicators in this case include two variable- views and post that are accessed through “Report”, a tool provided in a Learning Management System (LMS). The Student “Activity Logs” are also used as indicators of resource usage.
4. Measure (M): How to measure them? What data set can be used?
In this case study the mean, correlation and other statistical measures are calculated using SPSS. In addition, a limited number of Process Sigma calculations are also made using the available data set.
C. Application of Six Sigma approach for improving an eLearning course- A Case Study
For the purposes of this study an ICT course offered
in semester 2/2009 for M.S. in ICT
(www.eLearning.au.edu) program is considered. This course is offered as an eLearning course. Thirteen (13) students enrolled for this course. Typically, a eLearning course consists of many learning objects (LOs) or components which includes on-line video of a lecture for every unit, an audio file in MP3/MP4 format, a color or black and white pdf file of the Power-Point presentation, assignments, quizzes, forums, additional downloads and other components. In addition to these items there are other facilities that students use.
In a LMS a learning object (LO) or a component is usually defined as any entity, digital or non-digital that may be used for education or training. It is also called as web-based interactive chunks or parts of eLearning or courseware designed to explain a stand-alone training concept [6]. As a complement, a learning object (LO) or component should also have a measurable component of information which helps its identification, storage, and recovery through a database. For this research work the weekly “Reports” which provide access to “Activity Logs” in a LMS are identified as a rich repository of data which also constitutes the core of this work.
In its menu a LMS such as Moodle or Angel (Fig. 1) provides a tool called “Reports” which captures the conference data that can be used to analyze the pattern of usage of various LOs or components in a eLearning courseware. Other two important indicators- views and posts obtained from the reports are also used as indicators of what is happening in the Virtual Learning Environment (VLE).
III. RESULTS
In any eLearning courseware it is expected that a student will access the class website and use the given learning object (LO) or component at least once. In other words, by default every student should use the core components at least once. To begin with let us start with the 16 week data about two variables- views and posts.
Details of the views and posts for a sample ICT class for Semester 2/2009 used for this study is given in Table-1. Other statistical outcomes such as the correlation values for the two variables view and posts are listed in the last column of the Table 2.
The same data is also graphed in Fig. 3. Careful examination of this graph indicates a very low activity in terms of posts. It also indicates uneven activities in terms of views. However, the beginning and the end phase of the semester shows increased activity. In other words, in the sixteen week duration there are 4-6 weeks where there is increased interactivity with the LMS and the learning process. For rest of the weeks there is not much action. Application of Six Sigma methods can help in analyzing the reasons for the lack of activity during the course of studies.
TABLE 1
LOGS- VIEWS & POSTS
ICT-4 : Semester 2-2009
Period Views Posts
19-Dec-09 71 0 12-Dec-09 78 1 5-Dec-09 159 11 28-Nov-09 256 17 21-Nov-09 362 17 14-Nov-09 111 3 7-Nov-09 95 1 31-Oct-09 63 0 24-Oct-09 46 0 17-Oct-09 57 0 10-Oct-09 117 0 3-Oct-09 211 3 26-Sep-09 292 4 19-Sep-09 227 23 12-Sep-09 61 0 5-Sep-09 10 1 Total 2216 81
The correlation values for views and posts ranging between 0.743487 and 1.0 indicate a positive relationship. But standard deviation of 7.5 indicates wide variance in the usage of the various LOs or components provided in this courseware.
TABLE 2
CORRELATION & SUMMARY STAT
Correlation Matrix Views Posts Views 0.743487 Posts 1.0 Summary Mean 138.5 5.0626 StDev 7.5496
Fig. 3. Views & Posts activity Graph
In Table 3 a total of twenty two (22) learning objects (LOs) or components available in the sample eLearning courseware are identified. It also shows the usage of various LOs during the semester. Frequency (f) and percentage usage (%) of these LOs over the span of 16 week shows a wide range. This courseware consists of 12 chapters. Each chapter of the courseware has at least six core LOs or components. For a better understanding of the usage pattern of LOs percentage usage of various LOs is calculated and shown in the last column.
TABLE 3
LEARNING OBJECTS
Learning Objects (LOs)
Action Label f %
assignment upload LO1 14 0.56
assignment view LO2 93 3.74
assignment view all LO3 42 1.69 blog view LO4 5 0.20
course recent LO5 34 1.37
course view LO6 1075 43.19
discussion mark read LO7 4 0.16 forum add discussion LO8 7 0.28
forum add post LO9 32 1.29
forum delete post LO10 8 0.32 forum subscribe all LO11 2 0.08 forum update post LO12 9 0.36 forum user report LO13 6 0.24
forum view discussion LO14 208 8.36 forum view forum LO15 148 5.95
forum view forums LO16 56 2.25
resource view LO17 562 22.58
resource view all LO18 14 0.56 upload upload LO19 14 0.56
user update LO20 2 0.08
user view LO21 96 3.86
user view all LO22 58 2.33
Total 2489 100
Careful examination of the twenty two LOs listed in the table shows that LO6, LO14, LO15 and LO17 have the maximum usage during the semester. The action on LO17 “resource usage” is maximum (562/2489= 22.58%) compared to other objects. As expected, a student enrolled in an eLearning course should have a minimum of one instance of using each LO. However, in practice a student should use all the LOs or component more than once to master the concepts covered in a chapter or unit [7]. Six Sigma approach helps the instructor to locate the discrepancies between the expected usage count and the actual usage count available through the logs. The difference between the expected usage versus the actual usage of LOs or components is at the core of this case study. In Six Sigma “quantity drives the quality.” Applying this quantitative approach to usage of LOs or
components can definitely improve the quality of an eLearning courseware.
A. Applying Capability Maturity Model (CMM) to eLearning The next question is how the Capability Maturity Model (CMM) can be adopted for an eLearning program as a instructional technology solutions [8]? The answer to this question is as unique and customized as the program making these choices. The successful implementation of CMM in eLearning may involve folowing steps:
A clear identification of instructional problems or needs through data
A review of available tools; an adoption strategy; the adoption process based on Six Sigma results
A continued support and ongoing assessment. Throughout, the implementation approach should be periodically reviewed. The teaching and learning environment is alive and dynamic, as should be the solutions. The usage data should also emphasize various solutions.
In eLearning collecting data is important not only during the course production processes but also in the delivery phase. In conjunction with faculty members, the program should identify the instructional challenges that will be addressed and track whether or not the technology has been effective in meeting its goals [7]. Adoption of new solutions often evolves with support and experience; collecting data and analyzing it with Six Sigma approach can help to capture trends in usage of various LOS or components.
B. Analysis of data derived from system reports
The LOs data given in the Table 3 is also graphed in Fig. 4 given below. It shows an uneven use of various objects. As discussed earlier, the LO6, LO14, LO15 and LO17 have the maximum count. LO6 “course view” of 43.19% is about students logging on to the class website. It also means that the rest of the 56.81% activities cover usage of all the other LOs after the student is logged on to the system. The wide variation in the usage of various LOs or components should be a cause for concern. And the ways to rectify the concerns is to take strategic actions.
Fig. 4. Usage of learning objects (LOs)
Since the sample data for this study is very low the researchers take a partial view of how to apply Six Sigma in the given situation and suggest a process for improving the usage of LOs or components in a courseware. For the sake of discussion the details of the usage of LO17 alone are calculated and shown in Table 4. The last column in this table indicates 60% usage for LO17. The actual log (Table 3) from the LMS reports shows only 22.58% usage which is about one third of the expected. In other words, focusing on just one LO or component shows that LO17 had very low usage in the total learning process. LO17 represents a core learning resource which in this case includes on-line videos, audios and the power point files. In Six Sigma low usage of a product could be seen as something happening due to “defects.”
TABLE 4 EXPECTED USAGE OF LO17
Further analysis of each LO can yield some more interesting results. These results can be combined with data derived from views and posts for a better understanding of usage of various LOs or components of an eLearning or an eTraining program. Using Six Sigma terminology we can assume that there are “defects” in these LOs or components. Major causes of under usage could be attributed to defects which may be translated into lacking attributes of quality. Like any products or a service attributes of quality of an LO or component also needs further investigation.
Then the next question is- what should be done when the data indicate under usage of LOs or components in a courseware? Six Sigma process calculations for LO17 are given in Fig. 5. A simple Six Sigma calculation of LO17 shows Defect per Million Opportunities (DPMO) 399,573 which is about 39.96%. In other words, the low usage of LO17 will require a strategic review and a action plan to modify it for better usage. Further application of Six Sigma Root Cause Analysis (SSCA) may help design a course of action based on the following question.
Process Sigma Calculations for LO17 Opportunities (expected usage) = 936 Defects (actual usage) = 374 (936-562)
Calculation Results DPMO = 399,573 Defects % = 39.96 Yield % = 60.04 Process Sigma = 1.75
· What attributes of quality of a LO or component are defined or not defined?
· Was a cause-and-effect diagram used to explore the different types of causes (or sources of variation)?
· What tools were used to narrow the list of possible causes of under usage of LOs?
· Were charts (or similar tools) used to portray the 'heavy hitters' (or key sources of variation)?
IV. CONCLUSION
In early years of Six Sigma, the focus was on improving product quality. The quality assurance through Six Sigma requires assessment of all the variables involved in the eLearning courseware production process. This sample study covers just one course and its core components. Although limited, this work indicates that the eLearning courseware development cycle should incorporate Six Sigma process. As shown in Fig.2 the first step in the courseware development process should start with measurement goals. Traditionally, this critical step has been missing in the academic courseware development cycle.
The Six Sigma process discussed here is based on different types of measurements collected in LMS logs during a semester, and makes use of system reports and logs to create controls. It is thought to be used in evaluating the quality based on usage of LOs. Six Sigma process requires huge amount of data over a long period of time to better understand the “defects” in the components to sustain improvement. But it can be applied in a selected manner. Defects (identified here as non-usage or under non-usage of LO17 in the range of 39.96% is very high. In this respect, one important aim of this preliminary work is to establish use of Six Sigma evaluation methodology, and to gather more data from all sources for more comprehensive action. Another purpose of introducing Six Sigma in eLearning platforms is to establish some benchmarks for the indicators that can measured to improve the quality of learning objects (LOs) or components. Quality assurance in a eLearning programs is a growing concern among various stake holders, especially the students.
The acceptance of eLearning for the borderless education market requires that besides the quality of contents in a courseware all other related products and services should also continually improve [9]. Within few years eLearning and mLerarning will turn into a major eBusiness which will require more sophisticated methods for producing quality learning products and services.
ACKNOWLEDGMENT
Authors thank Dr. Brother Bancha Saenghiran, President, Assumption University for providing funding for this research work. Thanks are also to the CEO of the College of Distance Education (CIDE) Prof. Dr. Srisakdi
Charmonman and Dr. Chaiyong Brahamawong, Director, PhD program at CIDE for their consistent support and guidance.
REFERENCES
[1] Jones, S, The Internet Goes to College: How Students Are Living in the Future with Today’s Technology. Pew Internet & American Life Project. October 15, 2009. http://www.pewinternet.org/reports/toc.asp?Report=71
[2] Sener, J, Improving Access to Online Learning: Current Issues, Practices, and Directions. In: Bourne, J., and Moore, J. (Eds.), Elements of Quality Online Education: Practice and Direction. Sloan-C, Needham, MA, 2003
[3] Burge, S. Keith Hargrove & Legand., “Developing a Six Sigma methodology for improving retention in Engineering Education." in 32nd. ASEE/IEEE Frontiers in Education Conference, Boston, MA, USA, 2002
[4] Wiklund, Hakan Wiklund and Pia Sandvik., "Widening the Six Sigma concept: An approach to improve organizational learning." Total Quality Management, 2002, 13(2), 233-239.
[5] Robert E. Park, Wolfhart B. Goethert & William A. Florac, Goal-Driven Software Measurement-A Guidebook Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 15213 , August 1996, pp. 54-55
[6] M. Akrich, “The Description of Technical Objects,” Shaping Technology, Building Society, W. Bijker and J.Laws, eds, MIT Press, 1992, pp. 113-224.
[7] Lynch, Richard L. & Cross, Kelvin F, Measure Up!. Cambridge, Mass.: Blackwell Publishers, 1991.
[8] Mircea Diaconescu, Sergey Lukichev, and Adrian Giurca, ‘Semantic Web and Rule Reasoning inside of E-Learning Systems’, in Proceedings of 1st International Symposium on Intelligent and Distributed Computing, eds., C. Badica and M. Paprzycki, Studies in Computational Intelligence, Craiova, Romania, (18-20 October 2007). Springer.
[9] C. Adiele and W. Penner. Monitoring interactivity in a webbased community. In The Proceedings of the 7th International Conference on Web Information Systems Engineering (WISE06), 2006.