Chapter 5 : Case Study and Implementation
5.5 System Implementation
5.5.2.3 Assessment Rules
For the Assessment task, the student should be able to select the appropriate concept or method
among different methods. It provides an answer to the question “Why would you do that?”.
Furthermore, the student is able to justify the selection of a particular approach to solve a
problem. In this case, the system applies Familiarity rules as described in section 5.5.2.1 and
Usage rules as described in section 5.5.2.2 for each method or concept. The content produced
after applying these rules will help the learner to select the appropriate method or concepts
among different methods. Table 5-3 shows the rules that are used in the APELS system for
extracting significant key phrases and keywords from documents, in order to decide which
website satisfies the Familiarity, Usage, or Assessment tasks.
Table 5-3 Rules used to identify Familiarity, Usage, and Assessment tasks
Tasks Examples of
learning outcomes Rules
Familiarity
Learner understands what a concept is or what it means? Identify and describe uses of iteration 1. POS (NN+VB) 2. POS (NNS+VB) 3. Dependency relation (VB, NN) 4. Dependency relation (NN, VB) 5. POS (NN) 6. POS (NNS) Usage
Learner is able to use or apply a concept in a concrete way
Write program that use iteration
7. Dependency relation (VB, NN) 8. Dependency relation (NN, VB) In addition, Extract instance keyword pattern.
Assessment
Learner is able to justify the selection of particular approach to solve a problem
Determine which methods of
iteration is best for given problem
120
5.5.2.4 Implementation
The learning outcome validation process was added to ensure that the selected websites will
enable learning according to the learning outcomes set by standard curricula. The APELS
system produced a list of websites for “Fundament Programming Concepts” and “Algorithms
and Data Structure” and those with the highest similarity were chosen (see section 5.5.1.3).
Now, the appropriateness of the content of these selected websites will be assessed against
learning outcome as described by ACM/IEEE Computing Curricula.
As a first starting point to the topic, it is important for the learner to gain basic knowledge about
the topic to be able to answer the question: what do you know about it? This is usually achieved
by providing enough definitions, descriptions and/or examples, which is named as Familiarity
tasks in the APELS system. A score was given to the task based on the number of occurrence
of key phrases related to Familiarity in the content. The key phrases include the number of
definitions of topic; action verbs related to Familiarity tasks; topic names and their
relationships; as well as important terminology which helps the learners to fully comprehend
different concepts like the word “for example”.
As an example, the learning outcomes “define and describe the variable” for the “Fundamental Programming Concepts” module was assessed with Familiarity. In this case, the APELS
system, will select a set of URLs from different websites with a topic named “variable” being included in these URLs. Then the system applies three rules that were described in Familiarity
rules in section 5.5.2.1 to extract a number of significant key phrases and keywords from the
content. As some learners might prefer the topic to be explained practically with more
examples rather than being stated simply as definitions, it was important to use two rules that
were described in the Usage rules in section 5.5.2.2 to extract usage-related key phrases from
121
Table 5-4 Results of keywords and key phrases extraction that satisfy Familiarity for the learning outcome “Define and describe variable”
Table 5-4 shows that the system ranks the websites according to the Familiarity score, which
was calculated as a number of occurrences of keywords and key phrases in the content. The
results shown in Table 5-4 indicate that the website
(www.cplusplus.com/doc/tutorial/variables) occupies the first rank (25) because it has the
highest number of keywords and key phrases, extracted in the content that satisfies the
Familiarity learning outcome. Despite the importance of abstract concepts and explanation for
understanding the topic (variable), practical example is another key element, which was
included in the system evaluation. In this regard, the same website sustains a good score (21)
for usage task. On the other hand, the website (www.penguinprogrammer.co.uk/c-beginners-
tutorial/variables) occupies the fifth rank according to the Familiarity because it contains a low
score (8) of concept-related key phrases. However, it contains high score (29) according to
concrete/practical aspect, which is reflected by a high number of keywords and key phrases
related to usage and implementation of the knowledge around the variable topic. This particular
website might be of good quality content for learners, who prefer fewer definitions and more
practical examples.
WWW
abstract concepts and explanation Concrete, practical -POS (NN+VB)
-Dependency relation (VB, NN) -POS (NN)
Dependency relation (VB, NN)
Extract Keyword Pattern
Occurrences www.cplusplus.com/doc/tutorial/variables/ 25 21 en.wikibooks.org/wiki/C_Programming/Variables 19 23 www.tenouk.com/Module2.html 12 3 www.tutorialspoint.com/cplusplus/cpp_variable_types.htm 11 8 www.penguinprogrammer.co.uk/c-beginners-tutorial/variables 8 29 www.c4learn.com/cplusplus/cpp-variable-naming/ 7 2 cprogramming.pickatutorial.com/variables_datatypes.htm 6 4 www.tutorialcup.com/cplusplus/variable-types.htm 5 4 www.w3schools.in/cplusplus/variables/ 3 3
122
On the other hand, the other websites show poor rankings, as the number of keywords and key
phrases extracted is low. For example, the website (www.w3schools.in/cplusplus/variables) has
only 3 keywords and key phrases, which is not sufficient to select it based on the learning
outcomes which is referred to as “Familiarity”.
The second example of a Familiarity learning outcome used with the Algorithms and Data
Structure module is: “Discuss the importance of algorithms in the problem-solving process”. A learner often advances when the learning content provides key phrases that help him/her to
be aware with concept. These key phrases are defined and evaluated in our APELS system as
a Familiarity task. A score was assigned to the task based on the frequency of these key phrases
in the content of each website. Based on the learning outcome mentioned earlier in this
example. The same methodology described in the first example was applied to extract the key
words and key phrases related to Familiarity.
Table 5-5 Results of keywords and key phrases extraction that satisfy Familiarity learning outcome “Discuss the importance of algorithms in the problem-solving process”
The results (Table 5-5) showed that according to the Familiarity score the website
(staff.ustc.edu.cn/~csli/graduate/algorithms/book6/chap01.htm) occupies the first rank (29)
because it provided the highest amount of keywords and key phrases, extracted in the content
that satisfies the Familiarity learning outcome “Discuss the importance of algorithms in the
WWW
abstract concepts and explanation Concrete, practical -POS (NN+VB) -Dependency relation (VB, NN) -POS (NN) Dependency relation (VB, NN)
Extract Keyword Pattern
Occurrences staff.ustc.edu.cn/~csli/graduate/algorithms/book6/chap01.htm 29 4 www.personal.kent.edu/~rmuhamma/Algorithms/MyAlgorithms/intro.htm 15 2 en.wikibooks.org/wiki/Algorithms/Introduction 14 2 http://www.tutorialspoint.com/data_structures_algorithms/algorithms_basics .htm 12 3 www.macs.hw.ac.uk/~pjbk/pathways/cpp1/node32.html 5 1
123
problem-solving process”. Contrastingly, a low score was obtained with only 5 keywords and
key phrases from the website (www.macs.hw.ac.uk/~pjbk/pathways/cpp1/node32.html). As a
result, no understanding of algorithm concepts will be acquired by the students, as insufficient
data is provided to identify and comprehend the algorithms. Nonetheless, it should be
highlighted here that none of the websites give a good score regarding the concrete/practical
aspect of the content, however, that might be suitable for learners, who are trying to familiarise
themselves with the topic without going into the practical aspect of it.
After the learners have acquired the knowledge, they should be able to use or implement it
during the learning process. Using or applying knowledge in a concrete way is identified as
Usage tasks in our APELS system evaluation. The number of occurrences of the usage-related
key phrases such as “using a code to” or “implement variable” was found to get a score in order
to rank the websites based on the highest frequency of these key phrases. The key phrases
include the number action verbs related to Usage tasks; topic names and their relationships; as
well as the pattern syntax of instance keywords that relate directly to the language of the
programme. For example, a learning outcome “Write program that uses and implements s
Function” for the “Fundamental Programming Concepts” module was assessed with Usage. First, the APELS system will select a set of URLs, which contain a topic named “Function”.
Then the system applies two rules that were described in the Usage task (section 5.5.2.2) to
extract a number of key phrases from the content. As some learners might prefer to be provided
with more background information about the topic as well, it was important to use the three
124
Table 5-6 Results of keywords and key phrases extraction that satisfy Usage learning outcome “Write program that use and implement function”
The system ranks the website according to the Usage score. The results (Table 5-6) indicate
that the website (www.tenouk.com/Module4.html) occupies the first rank (29) because it has
the highest number of key phrases extracted in the content that satisfies the Usage learning
outcome. Meanwhile, same website has the highest number (31) according to abstract concepts
and explanation aspect, which is reflected by the high number of keywords and key phrases
related to understanding and defining the knowledge around the function concept.
On the other hand, the other websites (www.c4learn.com/cplusplus/cpp-functions-
introduction) and (http://en.cppreference.com/w/cpp/language/functions) show poor ranking,
as the number of key phrases extracted is low. For example, the website
(http://en.cppreference.com/w/cpp/language/functions) has only 2 key phrases. As a result, no
understanding of function concepts will be acquired by the students, as insufficient information
is provided to use and implement the functions. WWW
Concrete, practical abstract concepts and explanation Dependency relation (VB,
NN)
Extract Keyword Pattern
-POS (NN+VB) -Dependency relation (VB, NN) -POS (NN) Occurrences www.tenouk.com/Module4.html 29 31 www.penguinprogrammer.co.uk/c-beginners-tutorial/functions/ 23 12 www.programiz.com/cpp-programming/function 20 14 www.cprogramming.com/tutorial/lesson4.html 14 10 http://cprogramming.pickatutorial.com/functions.htm 12 13 en.wikibooks.org/wiki/C_Programming/Procedures_and_functions 10 18 www.tutorialspoint.com/cplusplus/cpp_functions.htm 7 19 www.learn-c.org/en/Functions 4 3 www.c4learn.com/cplusplus/cpp-functions-introduction/ 3 6 http://en.cppreference.com/w/cpp/language/functions 2 6
125
The second example of Usage learning outcome used with the Algorithms and Data Structure
module is: “Write a program that uses and implements an Array”.
Table 5-7 Results of keywords and key phrases extraction that satisfy Usage learning outcome “Write program that use and implement array”
The results (Table 5-7) show that the website (www.cplusplus.com/doc/tutorial/arrays) was
ranked first with a score of 26 because it has the highest number of key phrases, extracted in
the content that satisfies the Usage learning outcome. This high score indicates that this website
provides good material for a learner who aims to know how use the concept in a concrete,
practical way. In contrast, the website (https://ece.uwaterloo.ca/~dwharder/aads/Tutorial/1o/)
occupies the lowest rank according to the Usage task with a score of 4. However, it has a high
score at Familiarity, that might not be the best for students who had a good background about
the topic and looking for tools to apply their understanding in a more practical way.
The next level of mastery involves the ability of the leaner not only to use the concept, but also
to look at it from different viewpoints and explain why its approach to solve a certain problem
is ideal for him/her. This type of learning outcomes are defined as Assessment. As explained
before, a student in an Assessment task should be able to select an appropriate approach from
different alternatives. It provides an answer to the question “Why would you do that?” An example of an Assessment task that address the learning outcome in the programming concepts
WWW
Concrete, practical abstract concepts and explanation Dependency relation (VB,
NN)
Extract Keyword Pattern
-POS (NN+VB) -Dependency relation (VB, NN) -POS (NN) Occurrences http://www.cplusplus.com/doc/tutorial/arrays/ 26 22 cis.stvincent.edu/html/tutorials/swd/basic/arrays/index.html 17 20 www.tutorialspoint.com/data_structures_algorithms/array_data_structure.h tm 10 11 www.tutorialspoint.com/cplusplus/cpp_arrays.htm 9 7 https://ece.uwaterloo.ca/~dwharder/aads/Tutorial/1o/ 4 25
126
module was implemented in the system, which was “Choose appropriate methods of the
iteration for a given programming task”. In this case, Familiarity and Usage rules were applied
to assess the leaner knowledge around the topic. Then all the methods were identified for
iteration concept: for-loops, while-loops, Do-while and the system applied Familiarity and
Usage rules on each method. As a result, the student should understand multiple methods for
iteration and be able to appropriately select among them.
Table 5-8 Results of keywords and key phrases extraction that satisfy Assessment learning outcome “Choose appropriate methods of iteration for a given programming task “
In each website shown in the Table 5-8, all three type of iterations have been discussed, the
system ranked these websites as described before based on the highest score obtained with the
Familiarity and Usage tasks. Interestingly, the website
(www.cplusplus.com/doc/tutorial/Control) scored very high at both concrete/practical aspect
(30) and abstract concepts and explanation aspect (33) tasks suggesting it as a good content for
Assessment outcome to understand many aspects of iteration to be able to choose among all
different application available.
However, other websites obtained low score on either the practical or abstract concepts aspect.
For example, the website (www.penguinprogrammer.co.uk/c-beginners-tutorial/loops/) has a
low score of 4 at abstract concepts and explanation aspect, whereas, the website
WWW
Concrete, practical abstract concepts and explanation Dependency relation (VB,
NN)
Extract Keyword Pattern
-POS (NN+VB) -Dependency relation (VB, NN) -POS (NN) Occurrences http://www.tenouk.com/Module3.html 31 14 http://www.cplusplus.com/doc/tutorial/Control/ 30 33 www.penguinprogrammer.co.uk/c-beginners-tutorial/loops/ 27 4 http://cis.stvincent.edu/html/tutorials/swd/basic/control/repetition/index.html 22 31 http://www.cprogramming.com/tutorial/lesson3.html 18 11 https://cal-linux.com/tutorials/loops.html 16 16 https://turboc.codeplex.com/wikipage?title=LoopsCPP 9 27
127
(https://turboc.codeplex.com/wikipage?title=LoopsCPP) has 9 at concrete/practical aspect.
These low scores will lead to a failure in understanding what the topic is or how to use it.
Consequently, it will not be suggested to the learner with regards to the assessment task.
5.6 Summary
This chapter reviewed the ACM/IEEE Computing curriculum, which is an internationally
recognised and adopted standard in designing computer science related programs. In addition,
it described in details the body of knowledge of ACM/IEEE Curriculum which includes as
main components; the areas of knowledge, the subsequent units, required learning hours, and
learning outcomes. The second part of this chapter introduced the full implementation of the
APELS architecture using the field of computer science and the ACM/IEEE curriculum to
design two modules (Algorithms and Data Structure and Fundamental Programming
Concepts). All the components identified in the architecture were implemented and examples
were given to illustrate how they were applied in this specific case study. Finally, to evaluate
the learning outcome validation approach, the content of extracted learning material was
evaluated against a set of learning outcomes for the two modules were selected using this novel
128