2.2 The need of evaluation in e-learning process
2.3.2 The eight steps of SURE model
The SURE model consists of eight steps, see Figure 5. All steps of SURE model have a specific meaning. Output of the previous step will become the input of next step.
That are the eight steps of SURE model:
• Definition of key goals • Definition of sub goals
• Confirmation of evaluation goals • Creation of checklist
• Acceptance of checklist • Data collection • Data processing • The evaluation report
In the following we will explain the single steps of SURE model. Step 1. Definition of key goals
In the first step of SURE model, evaluation team has to define the key goals of evaluation. These goals are essential for main mission of evaluation.
B1 B2 B3 A11 A31 A12 A21 A22 a) b)
Figure 6: The logical structures of evaluation goals.
For example, we have to work on an actual running e-learning process. In this case we start with definition of key goals. These could be, for in- stance: Registration process (B1), Course material (B2) and Tutor skills
(B3). These key goals can be visualized by a logical series structure (see Fig-
ure 6a). These components were selected as key goals because without any registration learners have no right to access the e-learning framework. After successful registration learners can start the learning process. Without any e-content or the learning materials an e-learning cannot run. If the learners
are studying the e-materials without any tutoring, learners may loose their learning interest in e-learning and they can drop out from the study. The learners need tutor’s support and feedback during learning. All these facts show that these goals could be considered as key goals of evaluation.
Successful evaluation is then a result of:
• Administration • Course material • Tutor skills
If only one of these goals is failed, then the e-learning process will be evaluated as failed.
Step 2. Definition of sub goals.
During the second step, the key goals defined in the first step, have to be described in detailed manner if necessary. This can be done by sub goals. Sub goals can be understood as different ways to reach a key goal.
Example 1: To evaluate the performance of a registration process we
have two possibilities or ways as a rule: full online or blended registration. The corresponding key goal includes then two sub goals A11and A12which
refer to online and blended registration. The next key goal B2concerns the
course materials, for instance: this can be further divided into sub goals which refer to reading material quality (A22) and audio and video material
quality (A22). Course content can be designed in different ways: reading
materials which can be downloaded or read online; audio records which can be listened online; video materials which are included into course learning materials. Key goal B3could be focused to Tutor. In this case the key goal
B3is defined via a single sub goal A31.
Example 2: The e-learning process needs tutors. Tutor’s main task is
to support learners during e-learning process. But a tutor system is not an example for a parallel structure in the logical structure of an evaluation goal. An other example for parallel observation of an evaluation goal is considered Section 4.4.
Key goal Sub goal Question
B1 A11 Online registration process was easy to use
A12 Blended version of registration process was well
organized
B2 A21 Course content quality was high
A22 Course material level was high
B3 A31 Tutors support was very useful
Table 3: Checklist design proposal.
Step 3. Confirmation of evaluation goals.
The evaluation team has to confirm the final version of evaluation goals which have been defined during the previous two steps. If necessary, eval- uation team can define embedded or levelled logical structures until the evaluation goals are fully accepted by all members of the evaluation team. Further, the checklist for survey has to be adapted to the accepted logical structure. When all members of evaluation team agreeing to logical struc- ture of evaluation goals, that should be fixed by a protocol. This is helpful in avoiding conflicts between evaluation team and stakeholders.
Step 4. Creation of checklist.
The checklist is a well developed data collection method. There ex- ist many software solutions to create a corresponding online survey. For example: Monkey [99], fluidsurveys [100], iPerceptions [101], free online survey [102], kwik survey [103], easy polls [104], survey planet [105], Sogo survey [106], eSurveypro [107], esurvey creator [108], Stellarsurvey [109], Questionpro [110], esurv [111], questionform [112], panel place [113], sur- vey crest [114], addpoll [115] and Quick serveys [116].
For further discussion on this issue we refer to Penny [37], Betsy [7], Paul [8], Blayney [9] and Bridge [10]. The checklist of SURE model has to be created based on the sub goals of logical structure. Table 3 shows a proposal for checklist design for Examples. The generation of checklist can be supported by corresponding software.
However, existing software solutions cannot take reference to logical structure of the evaluation goal of SURE model. For the application of
SURE model we need a corresponding implementation tool. This imple- mentation should include functions like: checklist generation, data collec- tion and data processing.
Step 5. Acceptance of checklist.
Only an accepted checklist should be used for data collection. Clear for- mulation of questions is only one aspect of checklist. A further important aspect is the design of checklist. Each member of the evaluation team has to check that before confirmation and, if necessary, appearance and design of checklist has to be improved.
Step 6. Data collection.
There are several techniques for data collection: surveys and question- naire, tests and assessments, interviews, focus groups, action plans, case studies, and performance records [41]. Evaluation team can use any of these techniques. However, for the SURE model an online survey is rec- ommended. There is a beta version tool for SURE model (see Section 2.3). Via online survey data can be tabulated and processed automatically. This prevents human errors that usually occur during transferring the collected data to the data sheet.
Step 7. Data processing.
These are the main steps of data processing: Let us consider an evalua- tion structure C which consists of r key goals Bi, i = 1, ..., r, and each key
goal consists of sisub goals Aij, j = 1, ..., si, i = 1, ..., r. Then we have
C = r i=1 Bi = r i=1 si j=1 Aij.
We suppose that we have n checklist results obtained by an checklist adapt- ed to goal structure C. The evaluation interval for sub goal Aij be the
interval[x
ij, xij]. Let
x(1)11, ..., x(1)1s1, x(1)21, ..., x(1)2s2, ... , x(1)r1, ..., x(1)rsr,
...
x(n)11, ..., x(n)1s1, x(n)21, ..., x(n)2s2, ... , x(n)r1, ..., x(n)rsr,
swer of kthstudent to checklist question how the sub goal Aij has been
achieved.
Then the empirical score (computed score based on sampling results) that the aim of goal structure C has been achieved is calculated by
Q∗(C) = 1 n n k=1 Q∗(k)(C) = 1 n n k=1 r i=1 ⎛ ⎝1 −si j=1 1 − qij∗(k) ⎞ ⎠ . (2) Here denotes q∗(k)ij the empirical score for sub goal Aij according the kth
checklist result, k = 1, ..., n. It is obtained by normalisation of checklist value x(k)ij . It holds
q∗(k)ij = x
(k)
ij − xij
xij− xij .
Beside the empirical score Q∗(C) we can get estimation values for the key goal scores Q(B1),...,Q(Br) as well as for the sub goal scores Q(A11) , ...,
Q(Arsr). These are special cases in sense of formula (2). We get
Q∗(Bi) = 1 n n k=1 ⎛ ⎝1 −si j=1 1 − qij∗(k) ⎞ ⎠ and Q∗(Aij) = 1 n n k=1 q∗(k)ij .
The score values Q∗(C) and Q∗(B
i) are yet to transform by calibration
into the final empirical evaluation scores Q∗e(C) and Q∗
e(Bi). It holds Q∗e(C) = n1 n k=1 r r i=1 ⎛ ⎝1 − si si j=1 1 − q∗(k)ij ⎞ ⎠ (3) and Q∗e(Bi) = 1 n n k=1 ⎛ ⎝1 − si si j=1 1 − q∗(k)ij ⎞ ⎠ (4)
as well as
Q∗e(Aij) = Q∗(Aij). (5)
The evaluation scores allow a comparison of score values between different goal structures. The empirical evaluation score can be interpreted as an index of satisfaction of learners with an e-learning. A value Q∗e(Q) = 0.5 reflects an average level of satisfaction. Values over 0.5 an above average, values less than 0.5 a below-average satisfaction. The precision of obtained estimation values can be described by confidence intervals. For details we refer to Section 3.
Step 8. The evaluation report
By SURE model quantitative and qualitative results can be obtained. The outcome of first five steps forms the qualitative part of evaluation, steps 6 and 7 the quantitative part. The evaluation report has to include both parts of the evaluation process. The evaluation report has to make visible outcomes and findings of the evaluation process.
The kind of reports can be different depending on the audience to whom the report is addressed. By SURE model we can calculate several evaluation scores. Which evaluation score should be used for what, is to decide by the evaluation team. The graphics and charts are generated by SURE model application automatically, and these graphical representations can be used for the report.
Evaluation report has to be delivered to stakeholders or audiences at appropriate times. Else all effort of evaluation team could have been in vain. As a rule all interested groups expect a quick report after data collection.