FORMATIVE ADAPTIVE TESTING SERVICE WEB TO INDIVIDUALIZE E- LEARNING PROCESS

(1)

FORMATIVE ADAPTIVE TESTING

SERVICE WEB TO INDIVIDUALIZE

E-LEARNING PROCESS

NOUR-EDDINE El FADDOULI

Computer Science Department, RIME Team, Mohammadia School of Engineers (EMI), Mohammed Vth University Agdal, AV. Ibn Sina Agdal Rabat BP. 765, Morocco

[email protected]

BRAHIM EL FALAKI

Computer Science Department, RIME Team, Mohammadia School of Engineers (EMI), Mohammed Vth

University Agdal, AV. Ibn Sina Agdal Rabat BP. 765, Morocco [email protected]

MOHAMMED KHALIDI IDRISSI

[email protected]

SAMIR BENNANI

[email protected]

Abstract :

In most of the existing E-learning systems, the content and the order of activities are presented in a static manner without taking into consideration the learner’s characteristics. Therefore, customizing and adapting the environment to the learner’s profile has shown important improvement in education quality. There are several adaptation approaches of E-learning environments, such as, adaptive hypermedia system, semantic web, etc. In our proposed system (RIMEV), we adopted a Competency Based Approach (CBA), and we found that a well adapted system depends mainly on the adequacy of the information collected through a personal diagnostic. This diagnostic takes place via an adaptive test using the Item Response Theory (IRT) in a formative approach. This Intelligent test administers optimal items in an appropriate order in a short time and produces a good assessment of the learner’s skills with accurate results. Thus the e-learning systems can lead the learner to gradually acquire a skill by taking into account his/her characteristics and predispositions.

Concerning the technological aspect, the system is implemented as a composed web service, which calls several basic services that coordinate according to an orchestration plan. This allows interoperability with heterogeneous learning systems.

Keywords: Formative Assessment; Item Response Theory; Service Oriented Architecture; Learner Model, IMS QTI, Competency-based Learning

1. Introduction & context:

(2)

forward, it relies mainly on formative assessment and CBA to create an adaptive learning system. The purpose is to personalize the learning path through a customized diagnosis of the learner.

To implement the proposed system in its environment, several technical constraints arise. First, we model the competency and the learner according to CBA. Then, we consider assessment as an activity to incorporate into a learning unit. Finally, we design a bank of items (questions) while ensuring interoperability between heterogeneous systems. In our system, the enhanced approach of formative assessment [El Falaki et al, (2009)] will be adopted for a personalized diagnosis. Thus, the system offers a sequence of consecutive items. The answer to a given item determines the optimal item that should follow, taking into account the previous performances recorded in the learner’s model. The implementation of this process of adaptive testing is performed using Item Response Theory (IRT).

As far as the technical context is concerned, we adhere to our research team's vision [Khalidi Idrissi et al, (2009)] making the Service Oriented Architecture (SOA) the approach of integrating the e-learning framework. Consequently, the evaluation service is implemented as a composed service of several loosely coupled basic services, which will be orchestrated to regulate the learning process. To enable reuse and operability, the environment will be designed according to the existing standards, such as IMS-LD, IMS-QTI and IMS-LIP. The next section deals with the adaptive learning and tackles the personalization of E-learning path via the adaptive formative assessment in CBA. The third section concerns the item response theory to administer the optimal item and estimate skill level. The proposed system and its implementation will be presented in section 4, and we end up with a conclusion and research perspectives

2. Adaptive Learning System

The issue of personalization in E-learning has become an important field of research in recent years. Generally, personalization aims to adapt contents and services offered to the user to promote the quality of his interactions with the system [Stewart et al, (2004)]. In the education field, adapting is providing each learner with the feeling that the training is designed specifically to meet his/her expectations and his/her capacities.

Learning systems is a mediating tool among the learner, competency and other learning process actors. This mediation is the subject of several studies attempting to personalize it by taking into account the learner’s characteristics. The adaptation of learning is implemented through the adaptive learning systems.

In our proposal, we believe that adaptation is based on the identification of the learner, his/her ability, prior knowledge and current performance for the acquisition of competencies. Thus, we stipulate that two elements are essential, namely a learner modeling and a relevant diagnosis vis-a-vis the current activity. In this perspective, we modeled [Khalidi Idrissi, (2010)] formative assessment to offer the learning system a relevant diagnosis to regulate the learning process to each learner.

In this paper, it would be prominent to operationalize formative assessment by proposing an adaptive test which offers a selection of optimal items in a sequence taking into account the profile and the progression of the learner.

This implementation uses resources which integrate the concept of competency in a service-oriented approach. In this approach the different services (loosely coupled) are set up, and coordinate according to an orchestration plan.

2.1.Formative assessment in a competency based learning system

In the learning process, the role evaluation must not be limited to certification, but should be conceived in a formative approach to guide the teaching/learning process [Scallon, (2007)]. This approach was introduced by Scriven, M (1967) and supported by Bloom,B (1971). Formative assessment helps the student to learn; it participates in the regulation of the learning process [Perrenoud, (1998)]. It is also made up of a cycle that is built on three layers and that we enriched with a pre-regulation layer [El Falaki et al, (2010b)]:

Observation: Establish the position in relation to a repository, instead of confining the learner to a scale and comparing him/her to other learners.

Intervention: identifies symptoms to address the root causes of problems. It involves analyzing metacognitive knowledge [Perrenoud, (2005)].

Pre-regulation: it’s the step we proposed [El Falaki, (2010a)]; it offers optimal items to the learner. The purpose is to have a pertinent diagnostic allowing the regulation of the path learning for each learner

(3)

[IEEE, (2010)] and the IMS Reusable Definition of Competence or Educational Objective (IMS RDCEO) [IMS GLC, (2002)] specification. In our system [El Faddouli, (2011)], the competency has been defined in accordance with the IMS RDCEO specifications enriched by the HR-XML standard according to the recommendation of the European commission of normalization [CEN/ISSS CWA 15455, (2005)]

As far as the questions are concerned, to assess learners, items will be administered. This uses an item database structured in a formal and standard way. In our proposed system, we opt for the standard Question & Test Interoperability specifications (IMS-QTI), which allows representing the items data structure. Each of the items corresponds to a competency.

2.2.Adaptive Formative assessment:

In the majority of existing e-learning platforms, the questions given to learners and the sequence in which they are presented in an evaluation activity are the same. This raises various problems regarding the estimation of the skill level [Wainer, (2000)]. In such situations, the goal is to confine the learner to be on a scale and compare him/her to the other learners without taking into account his skill level. Indeed, the questions do not take into account the level of the learner and are administered randomly in a predefined order. Using this format of examination in a formative approach will not be of any use. Thus, we will opt for an adaptive test which is to provide each learner with questions tailored to his mastery of the subject, his profile and his answers to previous questions [Wainer, (2000)]. As far as his/her profile is concerned, a learner model is implemented according to the IMS LIP specifications

In a computing environment, the implementation of an adaptive test requires, first, we calculate the level of a given skill. Then, we use the learner’s model to decide the next question. Finally, we adopt mechanisms to select the items (questions)

In our proposal, to select items that will be administered to the learners, and to calculate the level of a given skill, we adopt a statistical probabilistic model named, the Item Response Theory (IRT)

3. The Item Response Theory (IRT):

Item Response Theory (IRT) was introduced to construct a formal approach to adaptive testing [Fernandez, (2003)]. IRT is generally regarded as an improvement over classical test theory (CTT). For tasks that can be accomplished using CTT, IRT generally brings greater flexibility and provides more sophisticated information. This theory aims firstly at estimating as accurately as possible le the learner’s skill based on his/her responses to items. Secondly, the evaluation of psychometric properties of items [Raîche et al, (2000)].

The model is based on a probabilistic mathematical representation described through a function linking the learner’s ability with the probability of a successful item. This function is called, the Item Characteristic Curve (ICC), which is the foundation of the IRT.

ICC represents for each item the probability P(θ) that an examinee with ability θ will give a correct answer to that item (fig. 1). The curve pattern depends on the item parameter values

Fig. 1. sample item characteristic curve

(4)

The Rasch model represents the structure, which data should demonstrate, in order to obtain measurements; i.e. it provides a criterion for successful measurement

In the (1PL) model (fig. 2), each item i is characterized by only one parameter, the item difficulty bi; this parameter shows a high correlation with the proportion of correct responses observed on an item. The model is known as Rash Model [Rasch, (1960)] and uses this parameter as follows “ Eq.(1)”:

Where D is a constant and equals 1.7 and Ө is the ability scale.

Fig. 2. Three-item ICC with different b values

Several assumptions are taken into account to make interpretations based on the TRI. The first is the heterogeneity of the variance: a latent trait unidimensionality, local independence from one item to another (the probability of getting a good response to an item is independent of the probability of getting a good response to other test items, invariance of the level of difficulty compared to subjects and an invariance of the skill level compared to the items

3.1.Constructing tests: Selecting the optimal item

Item information function (IIF) in IRT plays a central role in selecting optimal items to construct tests for examinees. Each item in a test provides information about the ability of the examinee. However, the quantity, quality and relevance of this information depend on how well the item’s difficulty corresponds to the learner skill level. The amount of information, provided through a single item, can be calculated for each item in terms of skill level according to the item information function “Eq. (2)”:

Where:

i is the item sequence number.

P (θ) is the first derivative of Pi (θ) and Qi(θ)=1- Pi(θ)

A test is composed of a set of items. Thus, for a given skill level, the test information is the sum of the item information at that level. Consequently, the test information function (TIF) is defined as “Eq. (3)”

Where Ii (θ) is the amount of information for item i at ability level h and N is the number of items in the test.

4. The proposed System:

4.1.System activities as a business process

A business process is a collection of interrelated tasks, which are designed to deliver a particular result

Today, e-learning systems are in need of integrating new pedagogical approaches that tend to offer better results and quality of learning. In order to implement such capabilities, the system logic will be considered as a business process. We opt for an SOA where we define the services and their interactions using an orchestration plan. For all the previously stated reasons, we had to adopt modeling methods in order to organize and combine

( ( ) )

1 ( )

1 i

i D b

P

e θ

θ = ₊ − −

(1)

Ii(θ)= Pi(θ) Qi(θ) (2)

1

( ) ( )

N i I

I θ I θ

=

(5)

these processes. There are several different ways to model a business process. Each has its own rules and syntax. This makes it difficult to work with models written in different languages. This is why a standard to represent business processes is critical at the modeling level as well as at the execution level. For our project implementation, we opted to use the BPMN [BPMN, (2002)] standard for business process modelling and BPEL [Rich Powers CSC 9010, (2008)] language at the execution level.

4.2.Orchestration

An important advantage of SOA is the Business process modeling. Business processes are modeled by the orchestration, which means that each service wouldn’t need to know about the other participating services in order to create a business process. Orchestration allows for each service to be independent and ensures that none of the participating service communicates with the other services. The orchestration handles calling services for execution in the frame of a predefined business process.

Service orchestration describes the matter in which services will interact with each other in order to create a business process. It includes the order of execution of the messages and the business process.

In the orchestration, the orchestrator is responsible for the composition and controls the interactions between services. This orchestrator coordinates in a centralized manner different operations of partner services. Business process description adds a view of the process and constitutes an excellent formalization and analytical tool to build systems. For such reasons, it is an essential component in information systems.

The orchestration allows reuse of services; it defines a plan to coordinate services according to the context, training, certification or skills assessment test (Fig. 3)

Fig. 3. reutilisation of services according to an orchestration plan

4.3.System Modeling:

4.3.1. Learner model based on IMS LIP

Learner modeling in CBA is a representation of the state of competencies. It should represent information characterizing the learner at the static level (profile) as well as at the dynamic level (progress). In RIMEV, the learner model is solicited in different activities. It can be implemented using standard templates. In our proposal, we adopt the IMS learner Information Package (IMS-LIP) specifications, which are “based on a data model that describes those characteristics of a learner needed for recording and managing learning-related history, goals and accomplishments” [IMS Global Learning Consortium, (2006)]

4.3.2. Items modeling based on IMS QTI

In the case of an evaluation activity proposed as part of a pedagogical scenario, items will be administered to the learner. This uses an item bank described in a standard way. Each item corresponds to a competency and will be calibrated in order to participate according to IRT in an assessment.

In the proposed system, we opt for the IMS-QTI standard, which allows representing the data structure of a question (item) and a test (assessment) and their corresponding results. This representation is done through an XML file providing interoperability

4.3.3. Competency modeling based on IMS RDCEO

(6)

competency in educational system. In our system, the competency has been defined in accordance with the standard IMS RDCEO enriched through HR-XML [El Faddouli, (2011)]

4.3.4. Modeling the system workflow

BPMN defines a Business Process Diagram (fig. 4), which is based on a flowcharting technique tailored for creating graphical models of business process operations. A business Process Model, then, is a network of graphical objects, which are activities (i.e., work) and the flow controls that define their order of performance. BPMN will also be supported with an internal model that will enable the generation of executable BPEL4WS. Thus, BPMN creates a standardized bridge for the gap between the business process design and process implementation.

For the implementation of a plan of orchestration, BPEL is recognized as a standard language for orchestrating Web services

4.4.Implementation System:

In the following section, we deal with the implementation of the four services used in the orchestration plan below (figure 4).

For each web service, an extract of the two files will be presented, namely the descriptive web service in accordance with WSDL that will be published in a register based on UDDI, and the internal logic file.

(7)

(1) observation Service:

The Observation service (fig. 5) establishes the position in relation to a repository, instead of confining the learner to a scale and comparing him/her to other learners. The internal logic of the service observation is modeled according to BPMN

Figure 5: Business Process Diagram for observation service For the implementation of this internal logic, the observation service in encoded as follows

Internal logic

External

Interface

(WSDL)

<System.Web.Services.WebService(Namespace:="http://tempuri.org/")> _

<System.Web.Services.WebServiceBinding(ConformsTo:=WsiProfiles.BasicProfile1_1)> _ <ToolboxItem(False)> _

PublicClass Service_observation Inherits ystem.Web.Services.WebService <WebMethod()> _

PublicFunction difference(ByVal valeur1 AsInteger, ByVal valeur2 AsInteger) AsInteger Dim gap AsInteger

gap = valeur1 - valeur2 Return gap

EndFunction EndClass

<wsdl:definitions targetNamespace="http://tempuri.org/"> <wsdl:types>

<s:schema elementFormDefault="qualified" targetNamespace="http://tempuri.org/"> <s:element name="difference">

<s:complexType> <s:sequence>

<s:element minOccurs="1" maxOccurs="1" name="valeur1" type="s:int"/> <s:element minOccurs="1" maxOccurs="1" name="valeur2" type="s:int"/> </s:sequence>

</s:complexType> </s:element>

<s:element name="differenceResponse"> <s:complexType>

<s:sequence>

<s:element minOccurs="1" maxOccurs="1" name="differenceResult" type="s:int"/> </s:sequence>

</s:schema> </wsdl:types>

<wsdl:message name="differenceSoapIn">

<wsdl:part name="parameters" element="tns:difference"/> </wsdl:message>

<wsdl:message name="differenceSoapOut">

<wsdl:part name="parameters" element="tns:differenceResponse"/> </wsdl:message>

<wsdl:portType name="Service1Soap"> <wsdl:operation name="difference">

<wsdl:input message="tns:differenceSoapIn"/> <wsdl:output message="tns:differenceSoapOut"/>

</wsdl:operation>

(8)

(2) Service pre-regulation :

In this step, the IRT will be used to select the optimal item. For this service we extract the WSDL file as follow:

<s:schema elementFormDefault="qualified" targetNamespace="http://tempuri.org/"> <s:element name="item_optimale">

<s:complexType/> </s:element>

<s:element name="item_optimaleResponse"> <s:complexType>

<s:sequence>

<s:element minOccurs="0" maxOccurs="1" name="item_optimaleResult" ty pe="s:string"/>

</s:sequence> </s:complexType> </s:element>

</s:schema> </wsdl:types>

<wsdl:message name="item_optimaleSoapIn">

<wsdl:part name="parameters" element="tns:item_optimale"/> </wsdl:message>

<wsdl:message name="item_optimaleSoapOut">

<wsdl:part name="parameters" element="tns:item_optimaleResponse"/> </wsdl:message>

<wsdl:portType name="Service1Soap">

<wsdl:operation name="item_optimale">

<wsdl:input message="tns:item_optimaleSoapIn"/> <wsdl:output message="tns:item_optimaleSoapOut"/> </wsdl:operation>

(9)

(3) Regulation:

This service describe the mechanisms that provide guidance, control and the adjustment of cognitive activities, that we modeled [ El Faddouli, (2011)]

5. Conclusion & perspectives:

In recent years, the issue of personalization has become an important field of research. Several approaches have been adopted. Ours is distinguished by the adaptive formative approach implemented, which personalizes the testing path to produce accurate results in order to regulate the learning process. In the implementation of the proposed system, interoperability and reuse justify the choice of components, the environment interacting with the system and the SOA as a framework

As a perspective, we intend to test our prototype in a pedagogical unit undertaken by a sample of learners. The results of the prototype implementation will be compared to those of the CTT in the same pedagogical unit

References

[1] A. Stewart, C. niederee, and B. Mehta, "State of the art in user modeling for personalization in content, service and interaction", DELOS Report on Personalization, NSF, 2004

[2] B. Bloom, Handbook on formative and summative evaluation of student learning, New York: McGraw-Hill, 1971

[3] B. El Falaki, M. Khalidi Idrissi, and S. Bennani, "Formative assessment in e-learning: an approach to personnalize diagnosis and adapt learning path", in IADIS e-Society 2010 (ES 2010) à Porto, Portugal, 2010, ISBN: 978-972-8939-07-6 pp : 391.395

[4] B. El Falaki, M. Khalidi Idrissi, and S. Bennani, "A Formative Assessment Model within competency-based-approach for an Individualized E-learning Path", in world academy of science, engineering and technology. Issue 64, april 2010, ISSN. 1307-6892, pp. 208-212

[5] B. El Falaki, M. Khalidi Idrissi, and S. Bennani," Modèle de l’évaluation Formative dans une approche par compétences pour un apprentissage à distance ", in Workshop sur les Technologies de l’Information et de la Communication’2009. Agadir. Maroc, 2009, ISBN :978-9981-0-2625-50

[6] Rich Powers CSC 9010, (2008) Service Oriented Architecture, Spring

<s:schema elementFormDefault="qualified" targetNamespace="http://tempuri.org/"> <s:element name="activite_optmale">

<s:complexType> <s:sequence>

<s:element minOccurs="0" maxOccurs="1" name="eff" type="s:string"/> </s:sequence>

<s:element name="activite_optmaleResponse"> <s:complexType>

<s:sequence>

<s:element minOccurs="0" maxOccurs="1" name="activite_optmaleResult" type="s:string"/> </s:sequence>

</s:complexType> </s:element> </s:schema> </wsdl:types>

<wsdl:message name="activite_optmaleSoapIn">

<wsdl:part name="parameters" element="tns:activite_optmale"/> </wsdl:message>

<wsdl:message name="activite_optmaleSoapOut">

<wsdl:part name="parameters" element="tns:activite_optmaleResponse"/> </wsdl:message>

<wsdl:portType name="Service1Soap"> <wsdl:operation name="activite_optmale">

(10)

[7] Business Process Modeling Notation, V1.1 http://www.omg.org/spec/BPMN/1.1/PDF

[8] CEN/ISSS cwa 15455, “A European Model for Learner Competencies”, ICS 03.180; 35.240.99, November 2005

[9] Fernandez, G. (2003) Cognitive scaffolding for a web-based learning environment, Proceedings of the 2nd International Conference on Web Learning ICWL’2003, Melbourne, 18-20, August 2003, pp. 12-20.

[10] G. Paquette, Modélisation des connaissances et des compétences pour concevoir et apprendre, Sainte-Foy: PUQ, 2002. [11] G. Scallon, L'évaluation des apprentissages dans une approche par compétences, Bruxelles: De boeck, 2007. [12] IEEE, CEN WS-LT LTSO, (http://www.cen-ltso.net/main.aspx?put=1054) 05/12/2010.

[13] IMS GLC, (http://www.imsglobal.org/specificationdownload.cfm), 2002.

[14] IMS Global Learning Consortium, Inc (2006a), IMS Learning Design Information Model, http://www.imsglobal.org/specificationdownload.cfm

[15] M. Khalidi Idrissi, F. Merrouch, and S. Bennani, "Analyse des situations e-learning : abstraction et modélisation" in 2nd Conférence internationale, systèmes d’information et intelligence économique, SIIE 2009. Hammamet, Tunisie, 12-14 Février 2009 ; IHE edition ISBN: 9978-9973-868-21-3. pp. 153-164.

[16] M. Khalidi Idrissi, B. El Falaki, and S. Bennani," Implementing the formative assessment within competency-based-approach applied in e-learning", 3d International conference on SIIE;; Sousse, TUNISIA; 18-20, 2010 . IHE edition ISBN: 978-9973-868-24-4, pp. 362-368;

[17] M. Scriven, "The methodology of evaluation". In Gredler, M. E. Program Evaluation. New Jersey: Prentice Hall, 1996. pp. 16. 1967 [18] N. El faddouli, B. El falaki, M khalidi idrissi, Samir bennani. Towards an Adaptive competency-based learning System using

assessment, IJCSI International Journal of Computer Science Issues, Vol. 8, Issue 1, January 2011 ISSN (Online): 1694-0814 [19] Ph. Perrenoud, Construire des compétences dès l'école, Paris: ESF, 2000

[20] Ph. Perrenoud, L’évaluation des élèves. De la fabrication de l’excellence à la régulation des apprentissages, Bruxelles: De Boeck, 1998 [21] Ph. Perrenoud, "Formation et Profession", Bulletin du Centre de recherche interuniversitaire sur la formation et la profession

enseignante, Vol. 11, n° 1, Montréal, avril 2005.

[22] Gilles Raîche, Richard Bertrand,Jean-Guy Blais 2000, Modèles de mesure: l'apport de la théorie des réponses aux items, Prese de l’université de quebec