D5.2: Report on first formative evaluation round

(1)

D5.2 Report on first formative evaluation round

METIS - Meeting teachers co-design needs by means of

Integrated Learning Environments

D5.2: Report on

first formative evaluation round

WP5: Evaluation

WP Leader:

<ITD-CNR>

Author(s):

Pozzi F. (ITD-CNR), Persico D. (ITD-CNR), Sarti L. (ITD-CNR), Brasher A. (OU), Chacón J.

(UPF), Dimitriadis Y. (UVA), Malatesta L. (KEK), Rudman P. (ULEIC), Serrano M. A. (Agora).

(2)

Project acronym: METIS

Project title: Meeting teachers' co-design needs by means of Integrated Learning Environments

Project number: 531262-LLP-1-2012-1-ES-KA3-KA3MP

Sub-programme or KA: KA3 Multilateral projects

Project website: http://www.metis-project.org

Reporting period: From 01/04/2013

To 28/02/2014

Report version: v.1.3

Date of preparation: 15/05/2014

Beneficiary organisation: University of Valladolid (UVa), Spain

Project coordinator: Prof. Yannis Dimitriadis

Project coordinator organisation: University of Valladolid (UVa), Spain

Project coordinator telephone number: +34 983 423696

Project coordinator email address: [email protected]

WP Leader: Francesca Pozzi (P5) WP Leader email address: [email protected]

Document history

Date

Version Author(s)

Description

13/03/2014

v.1.1

Pozzi, Persico, Sarti

Preliminary version

22/04/2014 v.1.2

Pozzi, Persico, Sarti

First version (revised according to

reviewers’ indications)

15/05/2014 v.1.3

Pozzi, Persico, Sarti

Second version revised according

to reviewers’ indications

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

This project has been funded with support from the European Commission. This publication reflects the views only of the author(s), and the Commission cannot be held responsible for any use which may be made of the information contained therein.

(3)

Executive Summary

The METIS project intends to contribute to the Technology Enhanced Learning (TEL) research field by providing educators with an Integrated Learning Design Environment (ILDE) and a workshop package for training them in using the ILDE, with the ultimate aim to foster and support effective learning design practices (METIS Consortium, 2012).

Work Package 5 (led by ITD-CNR) aims to help METIS in reaching its objectives, by carrying out an internal, ongoing evaluation of the two main outcomes (the ILDE and the workshop package), in order to fine tune and improve them.

The WP consists of three main tasks, namely:

 Task 5.1 “Development of assessment plan” – this task identifies the requirements for workshop and learning design enactment realization, as well as the evaluation

methodologies to be adopted.

 Task 5.2 “Evaluation of ILDE” – in this task the evaluation of the ILDE is carried out, taking into account the perceived ease of use and perceived usefulness by the system users.

 Task 5.3 “Evaluation of the workshop packs” – this task adopts a similar approach as T5.2, addressing the evaluation of the workshop packages and the enactment of the learning design with the involvement of the same set of users.

While in the previous WP5 deliverable (D5.1, submitted at month 5, i.e. March 2013) the overall evaluation strategy and plan were provided (output of T5.1), the present document (D5.2) reports on the first formative evaluation round (output of T5.2+T5.3), consisting of the development of the evaluation tools, carried out between month 5 and month 10 (August 2013), their use to gather data between month 11 (September 2013) and month 17 (March 2014) and the analysis of the data collected.

Main outcomes of this deliverable are thus the results of this evaluation process, which highlights the workshops package to be very well accepted by participants, even if some tuning to the format is still needed. The enactment phase needs to be further explored, possibly with more data in the second round and also the differences among the three sectors studied (i.e. higher education, vocational training, adult education) needs further attention. The ILDE was very positively perceived in all the contexts, even if the opinions concerning a real uptake of this innovative technology by the institutions are different. These results are used in the conclusive section to derive a list of implications for the second round of workshops, as well as a set of indications for the project work packages on the next actions to be undertaken.

(4)

D5.2 Report on first formative evaluation round 1. Introduction ...5 2. Evaluation in METIS ...8 3. Evaluation methods ... 11 Online questionnaire ... 13 Follow up interview ... 15

Structure of types A and B Interviews ... 16

Structure of Type C Interviews ... 18

Data tracked by the system ... 18

4. First evaluation in METIS - data analysis ... 19

Data from the workshops ... 20

Workshop at OU ... 20

Workshop at KEK ... 28

Workshop at Agora ... 34

Data from the enactments ... 41

Enactment at OU ... 41

Enactment at KEK ... 44

Enactment at Agora ... 49

5. Discussion and lessons learnt ... 50

Feedback on the workshops + enactments ... 52

Feedback on the ILDE ... 57

6. Conclusions and future work ... 59

7. References ... 63

8. APPENDIX I – Questionnaire ... 64

(5)

Project Number: 531262-LLP-2012-ES-KA3-KA3MP

5

1. Introduction

The METIS project has three main objectives (METIS Consortium, 2012):

1. To develop an Integrated Learning Design Environment (ILDE). The ILDE will

integrate existing free and open source solutions that include: co-design support for communities of practitioners; learning design authoring tools following different pedagogical approaches and authoring experiences; interface for deployment of learning designs on mainstream Virtual Learning Environments (VLEs).

2. To run a series of workshops for teachers at partner institutions using the ILDE. The workshops will be aimed at fostering the adoption of learning design methods among teachers and advancing their skills in the orchestration of ICT-based learning environments according to innovative pedagogical approaches. The ILDE will play a central role in the workshops, because one of the workshop goals will be to support teachers’ familiarization with the ILDE and to promote the usage of the tools integrated in it. The workshops will also envisage an enactment phase, i.e. a stage where a sub-set of workshop participants will have the chance to come back to their classes and deliver to students a design fully conceived, authored and implemented using the ILDE.

3. To disseminate the project’s outcomes and maintain a community of teachers engaged with learning design and its tools.

As described in D5.1 (Pozzi et al., 2013), METIS adopts a user-centred design approach: the development of both the ILDE and the workshops is cyclic, with two evaluation phases informed by practice (see Figure 1). These two evaluation phases (the former between the 2nd_{and the 3}rd_{cycle, the latter between the 3}rd_{and the 4}rd_{cycle), will incrementally}

(6)

6 Figure 1 - The METIS four cycles

The workshops are the basis for the formative evaluation of the different versions of the ILDE and of the workshop packs themselves. Thus evaluation plays a crucial role in the project, and occurs in an iterative and formative way, i.e. with the aim of informing the following stages of re-design and development.

Work package 5 (WP5) in the METIS project is explicitly devoted to evaluation. In particular, the main objective of WP5 is to carry out an overall evaluation of both the ILDE and the workshop package. This implies that WP5 in METIS is primarily focused on the evaluation of the two main outcomes of the project, and only indirectly addresses other aspects of the project evaluation.

The WP consists of three tasks, namely:

 Task 5.1 “Development of assessment plan” – this task identifies the requirements for workshop and learning design enactment realization, and the evaluation methodologies to be adopted.

 Task 5.2 “Evaluation of ILDE” – taking into account the perceived ease of use and perceived usefulness of the ILDE, a set of indicators and instruments to gather data are designed and developed in this task; various categories of the system users (end-users involved in vocational training, school teachers, higher education teachers, workshop conductors and organizers, etc.) are involved in the usage of the ILDE system.

 Task 5.3 “Evaluation of the workshop packs” – this task adopts a similar approach as Task 5.2, addressing the evaluation of the workshop packages and the enactment of the learning design with the involvement of the same set of user categories.

 Analysis of existing tools  Proposal of workshop pack

1

st

_cycle

 Release of ILDE prototype  Delivery of pilot workshop packs

2

nd

_cycle

 Second release of ILDE  Second delivery of workshop packs

3

rd

_cycle

 Final release of ILDE  Refinement of workshop packs

4

th

_cycle

EVALUATION

(7)

7 “D5.1 – Assessment plan”, which was delivered at month 5, contained the overall evaluation strategy (Task 5.1), by providing:

 an overview of the literature in the fields of: o evaluation of the impact of technology and o evaluation of training events;

 the specific theoretical frameworks adopted to evaluate the ILDE and the workshops in METIS;

 the definition of the main dimensions that are considered to evaluate the ILDE and the training events;

 a preliminary set of indicators to evaluate the ILDE and the training events;  the overall evaluation plan.

Given that D5.1 was planned so early in the project, at a time when both the ILDE and the workshop packages were still under development, it could not contain the evaluation tools, but only high-level indicators. Later on, when the project was able to provide details about the ILDE and the workshop packages, it was possible to define finer grained indicators and the actual evaluation tools (such as questionnaires, interview rubrics and tracking tools specifications).

“D5.2 - Report on first formative evaluation round” (Task 5.2 + Task 5.3), which is the present document, represents the continuation of the previous deliverable: it documents the first evaluation round (between the 2nd_{and the 3}rd_{cycle, see Figure 2), providing the}

description of the evaluation methods and tools designed and developed by the project, and the analysis of the gathered data, as well as the discussion about the main lessons learnt, which will inform the 3rd_cycle.

(8)

8 Figure 2 - Focus of D5.2

Figure 2 shows also D5.3 and D5.4, which are the following deliverables to be produced under WP5. In particular, while the former deliverable will contain the report on the second formative evaluation round, the latter document will provide a comprehensive view on the evaluation carried out during the project lifespan.

2. Evaluation in METIS

In this section we recall the main choices made by the project consortium concerning the evaluation and provide a synthesis of the overall evaluation model adopted in METIS (which is more extensively described in D5.1).

As already mentioned, the ultimate goal of METIS is the widespread adoption of innovative and effective learning design approaches and tools, by developing the ILDE and designing and running workshops for teachers on learning design using the ILDE. Consequently, specific METIS evaluation needs include:

 to measure acceptance of the proposed technology (ILDE) by the users (that means evaluating the extent to which users regard the ILDE as easy to use and useful);  to assess the adequacy and effectiveness of the workshops (and subsequent

enactment) in real contexts, by taking into account not only participants’ perceptions and reactions, but also the impact that the adoption of these innovations have on their institutions, as well as the contextual factors that may enhance or hinder their impact.

 Analysis of existing tools  Proposal of workshop pack

1

st

_cycle

 Release of ILDE prototype  Delivery of pilot workshop packs

2

nd

_cycle

 Second release of ILDE  Second delivery of workshop packs

3

rd

_cycle

 Final release of ILDE  Refinement of workshop packs

4

th

_cycle

EVALUATION

D5.1 D5.2 D5.3 D5.4

(9)

9 The latter point is particularly important to METIS, where three different contexts are being explored: vocational training, adult education and higher education (respectively

represented, in the project, by the three partners KEK, Agora and UKOU), since what may be shown as adequate in one context may turn out to be inadequate in another.

Thus, among the various existing models described in D5.1 to evaluate the technology, the TAM and its subsequent evolutions (in particular TAM2) have been chosen as theoretical framework to develop the core of the evaluation approach for the ILDE. Furthermore, the information provided by the application of this model will be complemented with data gathered from other sources, such as tracking data provided by the system itself. Thus, in METIS, user acceptance of the ILDE is measured mainly in terms of TAM’s indicators

“Perceived usefulness” and “Perceived ease of use”. Additionally, according to TAM2, social factors that influence ICT acceptance are also taken into account. In order to tailor the TAM to the METIS needs, and in particular to the ILDE evaluation, the approach takes into consideration on one hand, the ILDE functionalities defined in terms of “use cases” in Hernández-Leo et al. (2013), and on the other hand, the system components, as defined by the specification of the ILDE architecture (Pozzi et al., 2013).

As it will be further explained in the following, we chose the questionnaire as the main tool to collect evaluation information on both the workshop and the ILDE. As the evaluation of the ILDE intertwines with the evaluation of the workshops, data are collected in an integrated way through the same online tool, to minimize the effort of the

participants/users. In addition, the use of questionnaires complemented by appropriate guidelines for the workshop moderators, ensures that data collection is dealt with using the same procedure even when the evaluators are not present. The evaluation of the enactment activities further relies on interviews of participants.

In general, we strive to use appropriate triangulation processes to combine data that come from different sources and are collected using different techniques, such as log analysis, third-party observations, etc.

As mentioned before, the TAM and TAM2 provide information based on the users

perceptions and opinions, that is, subjective data that need to be complemented with more objective data about what actually happens when users engage with the ILDE. This

information is obtained thanks to tracking mechanisms in the ILDE, which provide, among other things, information on trustworthiness of the users’ opinions. If, for example, a user says that a given functionality was easy to use, but then the data tracked shows s/he never used it, he/she is less trustworthy than a user who claims the functionality was difficult to use after having engaged with it for a certain amount of time.

As far as the evaluation of the METIS workshops is concerned, Guskey’s model has been chosen as the main source of inspiration for the evaluation (Guskey, 2000; 2002), as it seems to have the closest fit with the project evaluation needs (see D5.1). This is because in his model, Guskey takes into account not only the participants’ perceptions and reactions to the

(10)

10 training event, but also the impact on the system (down to the level of students’ learning), as well as the contextual factors that may enhance or hinder the impact of the initiative. Both these aspects (user perceptions and impact on the system) seem particularly relevant in METIS, where the uptake and consequent diffusion of learning design innovative practices may meet obstacles and barriers once ‘transferred’ into real educational contexts, whose permeability to innovation is unknown.

According to Guskey (2002), effective teachers’ professional development evaluation requires the collection and analysis of five critical levels of information, namely: 1)

Participants' Reactions (it looks at participants' reactions to the workshops); 2) Participants' Learning (focused on measuring the knowledge and skills that participants gained during the workshop); 3) Organization Support and Change; 4) Participants' Use of New Knowledge and Skills; 5) Student Learning Outcomes.

Level 3 focuses on the organization: ‘lack of institutional support can sabotage any professional development effort, even when all the individual aspects of professional development are done right’ (Guskey, 2002). This means that the innovative practices that are the object of the workshops (e.g., in METIS the use of ILDE to support effective learning design), in real contexts, can fail to be widely adopted due to the contexts/systems where the workshop participants usually operate, which may not be ready to uptake the needed changes. Level 3 is important in METIS, given that we propose innovative practices to three different target users, operating in three contexts (Brasher and Mor, 2013), so we may discover that what fits the needs of one context is not in line with the policies of another one.

At Level 4 we focus on whether and to what extent participants are able to uptake the innovative practices proposed during the workshops and their ability to apply them within their daily practice. Unlike Levels 1 and 2, this information cannot be gathered at the end of the workshops, as enough time must pass to allow participants to adapt the new ideas and practices to their settings and enact them. In METIS a certain time is devoted to allow a sub-set of workshop participants to enact their innovative designs in their respective contexts (enactment phase, see D4.1 for an extensive description of this phase) and thus evaluation of Level 4 occurs after this phase (Rudman & Conole, 2014).

According to Guskey (2002), Level 5 addresses “the bottom line”: how did the workshop affect students/learners? Did it benefit them in any way? This means that after some of the participants of the workshops have implemented their designs, they are in charge of reporting what the impact of the innovation was on the student learning outcomes. By evaluating this level, which we indirectly address with the mediation of teachers, unintended (or unexpected) outcomes can be found.

As mentioned before, data collection of evaluation takes place at two different stages: data related to levels 1 to 3 are gathered at the end of the workshops (together with the data related to the ILDE evaluation) through the questionnaire and the analysis of logs, while data

(11)

11 related to levels 4 and 5 need more time and are evaluated after the enactment phase through interviews.

Furthermore, in analogy to what has been proposed for the ILDE evaluation, also in the case of the workshop evaluation the data coming from participants are compounded with other, more ‘objective’ data, coming from direct observation of what actually occurred during the workshop sessions themselves. This happens thanks to an observer, who takes notes during the workshops, supported by a rubric.

Observers also monitor and gather data during the enactment phases. All the data coming from observation, though, have been considered as part of the workshop enactment (WP4) and are thus extensively reported in D4.1 (Rudman & Conole, 2014). This deliverable takes these data as inputs and thus the final reflections about the results of the evaluation will draw also on them (even if they are not reported here). For this reason, if the reader wants to get a complete picture of the work done, it is recommended that the two deliverables are read one after the other (D5.2 after D4.1).

3. Evaluation methods

This section provides a summary of the evaluation tools designed, developed and delivered by the project under WP5 (see Figure 3).

Figure 3 - METIS events and evaluation tools

The two yellow circles in the Figure represent the two main events occurred during this first round in METIS, namely the workshop and the subsequent follow up phase, where the enactment took place.

All the other symbols in the Figure represent the evaluation tools developed, namely:

(12)

12  A questionnaire (blue circle) – delivered through an online form to all the workshop

participants at the end of each workshop; the questionnaire gathers data concerning:

o Profile of the workshop participant

o Participant’s opinions about the workshop (Levels 1, 2 and 3) o Participant’s opinions about the ILDE

o Free comments.

 A follow up bespoke interview (green circle) - to be delivered after the enactment phase to the workshop participants who agreed to enact; the interview gathers data concerning:

o Profile of the participant

o Participant’s opinions about the enactment (Levels 4 and 5) o Participant’s opinions about the ILDE

o Participant’s opinions about sustainability of the learning design innovations proposed

o Free comments.

Another, different bespoke interview is delivered to the workshop participant who did not agree to enact; this interview gathers data about the reasons behind their decision.

 Data tracked by the ILDE (red line) - these encompass the data related to the ‘actions’ performed on the platform by participants on learning designs during the workshops and the enactment.

 Data observed by observers (grey line) – reported in D4.1 (Rudman & Conole, 2014). Both the questionnaire and the interview included a consent form, where people should express their will about the use of data (see Appendix I).

The design of the questionnaire, the interview rubrics and the data to be tracked, are the result of an iterative process, coordinated by ITD-CNR with the contribution of all the partners, which started at month 3 of the project. Collaboration and discussion about these means occurred through emails and regular virtual meetings; besides, during the Barcelona meeting (July 2013) a dedicated session was organized for the partners to test the

questionnaire and for ITD-CNR to gather direct inputs and feedback about it and the overall evaluation strategy.

One of the main issues discussed with the partners is the degree to which it was advisable to ‘customize’ the evaluation means for the three different contexts addressed by the project (i.e.: vocational training, adult education, higher education). The METIS project had

developed a generic workshop package that was then adapted to the different contexts (see D3.3, McAndrew, Brasher, Prieto and Rudman, 2013). However, the differences introduced by the customization of the generic workshop package did not hinder the possibility to develop a common set of evaluation means capable of collecting the same categories of data from the different contexts, thus allowing for comparison of results across contexts.

(13)

13 There were, however, two kinds of customizations that were planned for and implemented where the three partners representing the different contexts deemed it appropriate. The first was the linguistic customization of the questionnaire: an equivalence table of the question items was produced to make it easy for partners to produce different versions of the questionnaire itself. This way, the questionnaire was produced in English by ITD-CNR and then translated only by those partners that deemed it appropriate (Agora translated it in Spanish, while KEK did not translate it in Greek since their end-users were comfortable with the English version). As for the interviews, these were carried out in the local language based on the rubrics (in English) produced by ITD-CNR. This method also left the interviewers free to adapt the questions to the context, where appropriate.

The second type of customization was related to the different ways learning design takes place in the three contexts; at KEK and Agora the teachers are also the designers of the training events, while the OU has a much more complex system featuring different actors in the design and teaching process and a longer time scale for the design-to-enactment

process. It was therefore necessary to customize the tools to assess the enactment phase for the different actors, according to a broad range of situations that emerged in the various contexts.

A further element of evaluation can be offered by the analysis of the artefacts produced by the participants during their activities. Rather than considering the explorative activities carried out during the workshops, when participants are concerned more on understanding the ILDE functionality than on delivering an authentic product, we mainly focus on the learning designs created during the enactment phase, which are to be delivered to actual students and must therefore feature completeness and self-consistency. These artefacts have already been reported in D4.1 (Rudman & Conole, 2014), so here we will take the elements emerged by D4.1 as inputs and source of further information.

The following sections contain an extensive description of each evaluation tool.

Online questionnaire

As mentioned above, two versions of the online questionnaire were produced: one in English (for KEK and OU) and one in Spanish (for Agora).

At the beginning of each workshop, the METIS project is introduced and participants are made aware of the fact they are taking part to a testing phase of the project (see D3.3, McAndrew et al., 2013). Thus participants are informed that, at the end of the workshop, they will be required to fill in a questionnaire and that their actions on the ILDE will be tracked during the workshop.

Given that it has been estimated that filling in the questionnaire would require about 20 minutes, the final 25-30 minute slot of each workshop is devoted to this activity.

(14)

14 In particular, through the questionnaire, participants are asked to provide the following information:

Data about the workshop participant. These include:

 Personal data (name, age, qualification, institution, position, etc.)

 Foremost background (mostly educational sciences; mostly computer science; educational technology; other)

 Work sector (academic: research/ university teaching; school teaching; industry; policy making; professional development; other)

 Previous knowledge about learning design (beginner, intermediate, expert, other)  Previous knowledge (if any) about any learning design tools among those integrated

in the ILDE

Participant’s opinions about the workshop (Levels 1, 2 and 3 of Guskey’s model). These include:

 Motivation – participants rate statements concerning their motivation to attend the workshop (Likert scale, from 0=low to 4=high)

 Reactions – participants rate statements about the quality of contents, quality of presentations, quality of discussion, effectiveness of hands-on activities, adequacy of the time schedule, adequacy of rooms and facilities, etc. (Likert scale, from 0=low to 4=high)

 Learning – participants rate statements about what they think they have learnt during the workshop (Likert scale, from 0=low to 4=high) and provide a self-evaluation of the outputs they have produced (namely the quality of the learning designs created during the workshop)

 Organization support & change - participants rate statements about the possibility that their institutions really adopt (some of) the innovations proposed at the workshop (Likert scale, from 0=low to 4=high). These data represent a preliminary investigation about the degree of sustainability of the innovation proposed by METIS  Free comments about the workshop (what did participants like most about the

workshop? What improvements should be done? etc.).

Besides the quantitative rating, participants can provide additional information in an open free text field (optional), in case they want to justify their ratings.

(15)

15 The structure of this section of the questionnaire reflects the structure of the ILDE1_{and thus}

collects users’ opinions about the following:

 Functions to create new learning designs – participants are asked to state if they understood what the ILDE offers and to rate ease and usefulness of such

understanding (Likert scale, from 0=low to 4=high). This includes 4 items.

 Conceptualizing functions - participants declare if they used the 9 conceptualizing functions and rate their ease of use and usefulness (Likert scale, from 0=low to 4=high). This includes 9 items.

 Authoring functions - participants declare if they used the 3 authoring functionalities and rate their ease of use and usefulness (Likert scale, from 0=low to 4=high). This includes 3 items.

 Implementing functions - participants declare if they used the 3 implementing functionalities and rate their ease of use and usefulness (Likert scale, from 0=low to 4=high). This includes 3 items.

 Browsing functions - participants declare if they used the 6 browsing functionalities and rate their ease of use and usefulness (Likert scale, from 0=low to 4=high). This includes 6 items.

 Sharing functions - participants declare if they used the 7 sharing functionalities and rate their ease of use and usefulness (Likert scale, from 0=low to 4=high). This includes 7 items.

 Free comments about the ILDE (what did participants like most about the workshop? etc.).

Besides the quantitative rating, participants can provide additional information in an open free text field (optional), in case they want to justify their ratings.

The complete questionnaires (in English and Spanish) are provided in Appendix I.

Follow up interview

As already mentioned, the different contexts in which the workshops took place determined the need for customizing the rubrics of the follow up interviews in order to take into account the different typologies of actors involved in the workshops (see D4.1, Rudman & Conole 2014).

In particular, three different rubrics were developed, namely:

A. An interview rubric for people who had attended the workshop and carried out the enactment within the timeframe required by the METIS project (2 trainers at Agora + 1 teacher at KEK).

B. An interview rubric for people who hadn’t attended the workshop, but carried out the enactment within the timeframe required by the METIS project (1 trainer at

(16)

16 Agora + 1 trainer at KEK). As it will be later on explained (see “Enactment at Agora” and “Enactment at KEK” sections), these are particular situations the consortium agreed to ‘take on board’, even if not originally planned, because they can contribute to the evaluation of this round.

C. An interview rubric for people who had attended the workshop but were not interested in carrying out the enactment (only at OU, sent to 15 workshop participants).

The first two interviews (type A and B) are carried out face-to-face by the partners

responsible for the respective context; ITD-CNR has provided the rubric and guidelines for the interviewers, so to guarantee homogeneity in data collection. At Agora and KEK the interviews are conducted in the respective languages and recorded; then each partner responsible translates the answers and provides them to ITD-CNR.

The third interview (type C) takes the form of a short written survey, consisting of a set of open-ended questions that OU circulated among its workshop participants via email.

Structure of types A and B Interviews

The first two types of interviews (A and B) have got the same structure, but B contains additional questions to compensate for the fact that participants who hadn’t attended the workshop, hadn’t filled in the questionnaire.

In the interviews, participants are asked to provide the following information: Data about the enactment participant.

These include:

 Personal data (name, age, qualification, institution, position, etc.)

 (for B only) Foremost background (mostly educational sciences; mostly computer science; educational technology; other)

 (for B only) Work sector (academic: research/ university teaching; school teaching; industry; policy making; professional development; other)

 (for B only) Previous knowledge about learning design (beginner, intermediate, expert, other)

 (for B only) Previous knowledge (if any) about any learning design tools among those integrated in the ILDE.

Participant’s opinions about the enactment (Levels 4 and 5 of Guskey’s model). These include:

 (for B only) Motivation – participants rate statements concerning their motivation to be involved in this experience (Likert scale, from 0=low to 4=high)

(17)

17  (for B only) Learning – participants rate statements about what they think they have

learnt during the experience (Likert scale, from 0=low to 4=high)

 Use of knowledge and skills during the enactment – participants are asked to give details and then provide free opinions about the following enactment phases:

o Creation of new learning designs during the enactment

o Completion/refinement of learning designs created at the workshop o Implementation of their learning designs in a VLE

o Delivery of learning designs to students o Evaluation of their learning designs.

 Student learning outcomes – participants are asked to give details and provide opinions about the delivery to students and the impact the learning designs might have on them and their learning. In addition, participants rate statements

concerning the impact on students (Likert scale, from 0=low to 4=high). Participant’s opinions about the ILDE.

Opinions are asked for each of the following sets of ILDE functions:  Functions to create new learning designs

 Conceptualizing functions  Authoring functions  Implementing functions  Browsing functions  Sharing functions.

This section is treated differently in cases A and B: participants who had attended the workshop (A), are reminded of the answers they had provided for this section in the online questionnaire, and are asked to express and comment their current opinions in the light of the enactment experience, in case their opinions have changed.

Participants who hadn’t attended the workshop (B) are asked opinions on the same functions, but in the same form of the questionnaire (see Section “Online questionnaire” above).

Participant’s opinions about sustainability

This section is explicitly devoted to sustainability and the data gathered here should complement those already gathered with the initial questionnaire (section “Organizational support & change”).

Participants are asked to rate statements concerning the perspective adoption at their institutions of both the ILDE and the workshop, and, more in general, the degree they think these innovations fit the needs of their work sector (vocational training, adult education, university). These data will support reflections about the possibility to further export these innovations to other, similar contexts.

(18)

18 Free comments

Participants are free to express any other comment/opinion (about the ILDE, the workshop, the enactment or even the evaluation means and procedures).

Structure of Type C Interviews

Coming to the third interview proposed to those who attended the workshop, but did not enact, the survey includes the following open text questions:

 Changes brought by the workshop to the everyday participant’s practice (if any)  Reasons for changes or no changes

 Plans for the future (in terms of changes).

The three complete interview rubrics are provided in Appendix II.

Data tracked by the system

The definition of the data to be tracked by the ILDE was the result of a negotiation between WP5 (Evaluation, led by ITD-CNR) and WP2 (ILDE development, led by UPF). During the negotiation, it was agreed that data would be defined following the “use cases” conceived during the design of the ILDE and described in D2.1 (Hernández-Leo, Chacón, Prieto, Asensio, and Derntl, 2013).

Thus, drawing from the use cases, the following data have been defined as essential for evaluation purposes:

 Choose a Tool

o Number of designs created with each tool by each user o List of designs created with each tool by each user  Produce a learning design

o Number of designs produced by each user

o Number of designs created / modified by days/weeks/months o List of designs modified by each user by days/weeks/months  Co-produce a learning design

o Number of reviews (edits) to a design (in global or by users) o Number of users editing a design

o List of users editing a design  Share a learning design

o Number of users with whom a design has been shared with editing right o Number of users with whom a design has been shared with view right o Users with whom a design has been shared with editing right

o Users with whom a design has been shared with view right o Number of designs shared with others with editing rights o List of designs shared with others with editing rights

(19)

19 o List of private designs

 Instantiate (Implement) a learning design

o Number of designs associated to at least a VLE (Virtual Learning Environment)

o List of designs associated to at least a VLE by implementer

o Number of implementations (design associated to VLEs) for a design  Deploy an instantiated learning design

o Number of times a deployment package is created for a design o Number of feedback provided on a learning design

o Number of comments associated to each design by commenter  Explore learning designs

o Number of times a user viewed a design (by user and by designer) o Number of times a design has been viewed (by user and by designer)

Additional data that were collected regarded information on the ILDE functions more related to the community building, namely:

o Number of members in the community o Number of designs produced by a member o Number of documents associated to each design o Number of comments associated to each design o Number of reviews to a design

o Number of users editing a design o Number of tags associated to a design

o Number of designs created / modified by days/weeks/months o Number of groups

o Number of members in each group

o Number of designs published (accessible outside LdShake) o Number of visits to the site / pages

o Number of designs created with each tool.

4. First evaluation in METIS - data analysis

The first round of the METIS workshops occurred in the three contexts chosen by the project (namely at OU, KEK, and Agora) in autumn 2013 with different time schedules and

modalities. The design of the three workshops is extensively described in D3.3 (McAndrew et al., 2013), while their running is described in D4.1 (Rudman & Conole, 2014). The enactment phase occurred in the early months of 2014 (again details about the enactment can be found in D4.1).

In this section (Section 4), the data gathered during the running of workshops and enactments are reported, through the evaluation means developed under WP5 and described in the previous section. Section 5 will discuss these data.

(20)

20

Data from the workshops

This section summarizes the information about the workshops that are needed to make this document self-contained. The section has three subsections, each devoted to one of the three workshops. The data reported about the running workshops mostly derive from the questionnaire analysis, complemented by the tracking data usually in the last column of the relevant tables.

Workshop at OU

Data about the workshop participants

The workshop at OU took place on 24 October 2013, and involved 17 participants, 13 of which (76%) filled in the questionnaire.

According to the data gathered through the questionnaire, all of the respondents work at the Open University, but they cover different roles: 2 librarians, 1 staff tutor, 4 lecturers, 2 media developers, 2 media project managers, 1 information literacy specialist, 1 assistant director.

Their backgrounds are diverse: 2 persons declare their foremost background is in

educational science, 1 in computer science, 1 in sociology, 1 in learning media design, 1 in media production/publishing, 2 in publishing and project management, 1 in management learning, 1 in learning development, 1 in English, 1 in library and information systems, 1 in health care.

As far as their previous experience on learning design, 3 persons (23% of the respondents) declare they are beginners in this field, while the others declare they are intermediate. Almost all the tools proposed at the workshop were previously unknown to participants (except for Course Map which was known by 4 persons) and 2 persons declare they know respectively Compendium and the Open University Learning Design Initiative (OULDI)2_tools

and resources.

When asked about the motivations which led them to attend the workshop, they were required to rate a set of statements (the labels of columns 2 to 5 in Table 1) from 0 to 4. Looking at the results obtained (see Table 1), it emerges that intrinsic motivation played a strong role for the workshop participants to decide to attend (“I am interested in learning deigns”, “I need to learn about learning design”); however, it seems that the social norms (“People I work with believe learning design is important”) influenced the decision even more. All the respondents declared they were not at all forced to enroll in the workshop (rate=0), except for two persons who rated respectively 2 and 4 this option.

2_{http://www.open.ac.uk/blogs/OULDI/}

(21)

21 Table 1: Motivation (OU participants)

Motivation (from 0=inessential to 4=very important) I am interested in learning design I need to learn about learning design People I work with believe learning design is important

I was forced to enrol in this workshop

mean 3,38 3,42 3,45 0,55

st. dev. 0,65 0,79 0,69 1,29

Participant’s opinions about the workshop (Levels 1, 2 and 3 of Guskey’s model). Looking at the evaluation data provided about the workshop, and in particular at participants’ reactions (Level 1), it seems that the most appreciated elements of the

workshops at the OU were the hands-on activities and the quality of discussion (see Table 2); the relevance of the workshop and its suitability with regard to participants’ prior

competence was pretty high, and also the quality of contents was positively evaluated. In contrast, the adequacy of the time schedule was less appreciated and even rooms and facilities were not completely satisfying. In particular most participants claimed that activities would have needed more time and a suggestion came to anticipate some of the activities (for example familiarization with CSCL, thinking at a possible design, etc.) before the workshop. One participant claimed that either the room was not big enough or enough spaces had not been allocated per subject desktop.

Table 2: Reactions (OU participants)

R e ac tion s (f ro m 0 =v ery lo w t o 4= ve ry h igh) Su ita b ili ty with r egard to yo u r p rior comp eten ce Re le van ce o f t h e w o rk sh o p co n ten ts Qu alit y o f p re se n ta tio n s Qu alit y o f t h e d is cu ss ion Ef fe ctiv en es s o f h an d s-o n a ctiv itie s Ad eq u acy o f t im e sch ed u le Ad eq u acy o f r o o m s a n d facilitie s mean 3,00 3,00 3,00 3,23 3,42 2,25 2,92 st. dev. 0,85 0,91 0,82 1,01 0,67 0,75 1,04

As far as the evaluation provided by participants regarding their learning at the workshop (Level 2), while they think they have learnt a lot, there is some uncertainty related to its application in daily practice and in particular their ratings about the possibility that this will speed up their learning design activities, are quite low (Table 3).

(22)

22 Table 3: Learning (OU participants)

Learn-ing (from 0=very low to 4=very high) I have learnt a lot from this workshop I look forward to putting into practice what I have just learnt about learning design I intend to use the ILDE in my future learning design activities Applying what I have learnt during this workshop will improve the quality of my courses What I have learnt in this workshop will speed up my learning design activities mean 3,00 2,92 2,23 2,77 2,00 st. dev. 1,00 0,95 1,09 0,83 1,00

Participants declared they have created (i.e. conceptualized or authored) a mean of 1,15 new learning designs each, using the ILDE, while only two persons declared they had implemented 1 learning designs each; this is because, due to time restriction, the hands-on activity originally planned for the implementation phase, was then turned into a

demonstration, so it is reasonable that not many people experimented it (see D4.1, Rudman & Conole 2014). In any case, we should also keep in mind that at the OU people worked in teams, so these figure reflect a group work, rather than an individual effort.

Then participants were asked to evaluate their most complete learning designs and the ratings they provided are contained in Table 4. Obviously, we should consider that these first creations are necessarily ‘explorative’ in nature, given also the limited time available during the workshop.

Table 4: Learning: ratings of the designs (OU participants)

Learning

(from 0=very low to 4=very high)

Completeness Originality Complexity Reusability

mean 2,00 1,83 2,00 2,58

st. dev. 0,60 0,72 0,85 1,08

As to the Level 3 (Organization support and change), there is a certain confidence that the ILDE and more in general the practices learnt at the workshop could be adopted by

colleagues, but more at an informal level, rather than at an institutional one: as a matter of fact, participants claim that institutions are hard to change and usually very few tools achieve widespread/full adoption at their institution (see Table 5). As it will be further discussed in this document, this perception may derive from the peculiar context of OU, as this organization already has a production process for the generation of e-learning material and activities in place.

(23)

23 Table 5: Organizational support and change (OU participants)

Org. support & change

What I have just learnt about learning design could alter the existing practices in my institution

The adoption of the ILDE as institutional tool to support learning design is likely to be fostered by my institution

What I have just learnt about learning design could be transferred to my colleagues ILDE could be adopted by my colleagues mean 2,38 1,73 2,75 2,55 st. dev. 1,19 0,90 0,75 0,93

Lastly, when asked to list three words to describe the workshop, participants expressed the following (see Figure 4).

Figure 4 - WordleTM_{created with the words expressed by participants}

to describe the workshop3_(OU)

As one may see from the Wordle, “interesting, stimulating and informative” are recurrent terms, but also “rushed” is a keyword which we need to take into due consideration. The most appreciated activities were the paper prototypes and the storyboarding, the most appreciated side-effect was considered the possibility to meet and discuss with colleagues. Suggestions for possible improvements concerned mainly a better timing, possibly

anticipating some activities and/or providing background readings before the workshop itself, or even spreading it over another session. Besides, a better preparation of the guidance materials and a better targeting to participants’ needs were also advocated. Participant’s opinions about the ILDE

As far as the evaluation of the ILDE provided by participants, Table 6 shows that OU

participants were not so positive concerning the ease of use and usefulness of accessing the

3_{The word cloud gives greater prominence to words that were used more frequently.} http://www.wordle.net/

(24)

24 various tools within the platform and understanding the differences among them.

Usefulness in the table is always higher than ease of use. Table 6: Functions to create new designs (OU participants)

Functions to create new LdS4

(from 0=min. to 4=max.)

how many say they did it ease of use (mean /st. dev.) usefulness (mean / st. dev.) understanding that the

ILDE provides access to a variety of tools 10 2,50 0,85 2,67 0,87 understanding the

differences between these tools

9 2,11

0,60

2,75 0,89 finding these tools in the

ILDE 8

2,00 0,76

2,57 0,79 access/enter these tools

from the ILDE interface 8

2,13 0,83

2,43 0,98

According to the tracked data, their exploration of the tools for conceptualizing the design was mainly focused on the Heuristic Evaluation tool (which was encouraged by the

facilitator), with some exploration of the Persona Card, the Factors and Concerns template and the Image upload (see column ‘tracked data of the Table below). The ease of use of the Heuristic Evaluation was rated on average 2,30 (st. dev. 0,67) and its usefulness 2,80 (st. dev. 0,79). Higher ratings were attributed to the ease of use and usefulness of the Persona Card5

(see Table 7). Unfortunately, no ratings are available for the Factors and Concerns function.

Table 7: Conceptualize functions6_{(OU participants)}

Conceptualize functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) Course Map 2 2,50 0,71 2,50 0,71 0

4_{In the ILDE terminology, “LdS” stands for “learning designs” (see D2.1).}

5_{The table shows that 4 persons expressed their opinions about the Persona Card (first column), but} then the data tracked by the ILDE say that only 1 LdS was created with this function (last column). This should not be regarded as ‘contradictory’ (and similar situations will be common also in the following tables); it may be caused by one of the following: a) a group of participants jointly explored one function (using one group account) and then all its members expressed their opinions about the function in the questionnaire; b) a number of people entered the function and rated it, but only a sub-set of them saved their LdS; c) one function was demonstrated at the workshop, so even if

participants didn’t experienced it directly, they rated it on the basis of what they had been shown. 6_{The description of the Conceptualize functions can be found in D2.1.}

(25)

25 Conceptualize functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) Design Pattern 3 2,33 0,58 2,33 0,58 0 Design Narrative 1 2,00 2,00 0 Persona Card 4 3,25 0,96 3,25 0,50 1 Factors and Concerns 0 -- -- 2 Heuristic Evaluation 10 2,30 0,67 2,80 0,79 10 CompendiumLD (upload) 0 -- 0 Image (upload) 3 2,00 1,00 3,00 1,00 3 For other conceptualizations 0 -- -- N.A. 7

As to the Authoring functions, the OU participants extensively experimented with WebCollage, given that 17 designs were created8_{(see last column in Table 8); they rated}

positively its usefulness, but they did not find it easy to use (Table 8). Table 8: Author functions9_{(OU participants)}

Author functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) WebCollage 8 1,88 0,64 3,00 0,58 17 OpenGLM 0 -- -- N.A. CADMOS 0 -- -- N.A.

Evaluation data about the Implementing functions are not very meaningful (see Table 9), given that no designs were implemented and only one person rated it (Implementation was demonstrated at the OU, see D4.1, Rudman & Conole 2014).

7_{N.A = not available; this means that the ILDE didn’t track this function.}

8_{To be noted that the 17 designs authored might include different versions of a designs aimed to the} same outcomes.

(26)

26 Table 9: Implement functions10 (OU participants)

Implement functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) Implement in a VLE through Glue!-PS 0 -- -- 0

See your VLE 1 2,00 3,00 N.A.

Register your VLE 0 -- -- 0

More informative are the ratings attributed to the Browsing functions: the most

experimented and appreciated ones being the “Free search” and the “Tag browsing” (see Table 10). Although not rated by many people, the data show very good results, especially as far as the Usefulness, but also the Ease of use of the Browsing functions.

Table 10: Browsing functions11_{(OU participants)}

Browsing functions

how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) Free search 5 3,40 0,55 3,60 0,55 Browse by tools 1 3,00 4,00 Search patterns 1 3,00 0,31 Browse by tags 4 3,50 0,58 3,75 0,50 Browse by discipline 1 3,00 4,00

Browse by pedagogical approach 1 3,00 4,00

As far as the Sharing functions are concerned, the few people who experienced them, judged them positively. Very likely, the reasons why few people managed to use these functions, have to do with the lack of time devoted to it during the workshops rather than difficulties in use (Table 11). It is reasonable to assume that Sharing functions will play a more relevant role in the follow up of the workshop, when people may feel the need to share their ‘real’ designs with others for mutual enrichment and exchange.

Interestingly, in one case (“Share a LdS with others with view rights”), even if it seems that many people explored this function, only one of them rated the corresponding function in the questionnaire. This may raise doubts about the clarity of the question.

10_{The description of the Implement functions can be found in D2.1.} 11_{The description of the Browsing functions can be found in D2.1.}

(27)

27 Table 11: Sharing functions12 (OU participants)

Sharing functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of actions (tracked data) Create a LDshakers' group 1 3,00 2,00 N.A. Share a LdS with others with view

rights 1 4,00 4,00

13 Share a LdS with

others with edit rights 0 -- -- 3 Add a comment to a LdS 0 -- -- 0 Exchange messages with other LDshakers 4

2,50 1,29

2,75

1,50 N.A.

View someone else's

LdS 1 3,00 2,00 N.A.

Edit someone else's

LdS 0 -- -- N.A.

Even if only five participants provided words to describe the ILDE, from Figure 5 it emerges that, while the platform is considered (potentially) useful, at this stage of development it is considered still unintuitive.

to describe the ILDE (OU)

According to participants, the most appreciated aspect of the ILDE is its ability to make people reflect on the design process, thanks to the rigorousness of the approach and the possibility to share designs with others.

As far as possible improvements, even if many participants declared it is too early to give an opinion on that, four of them declared the interface should be improved, so to make it

(28)

28 easier to use and navigate and two others suggested that the terminology / labelling should be improved.

Workshop at KEK

Data about the workshop participants

As it has been extensively described in D4.1 (Rudman & Conole 2014), the workshop at KEK took place on 9 November 2013, and involved 18 participants, 16 of which (the 89%) filled in the questionnaire. According to the data gathered through the questionnaire, they are either school teachers (in the field of chemistry, physics and biology) or teacher trainers, except for 1 school counselor. To be noted that, although KEK was chosen as representative of the vocational training sector, in this first round the enrolled participants do not belong to the expected category; this means that the data emerging from this workshop, which still remain of interest for the project as they provide useful feedback about the way the workshop itself and the ILDE can be perceived by school teachers, shouldn’t be considered as belonging to the vocational training sector.

As foremost background 14 declare educational science, and 2 computer science. As far as their previous experience in learning design, 12 persons declare they have been working in this field for a few years, while 3 declare they are experts (1 missing answer). Despite such a prior experience, all the tools addressed at the workshop were previously unknown to participants.

When asked about the motivations which led them to attend the workshop, they answered by rating from 0 to 4 the statements reported as headings of columns 2 to 5 in Table 12. The results indicate that intrinsic (personal interest / need) and extrinsic (social norms)

motivation played an equal role. Only one person rated 1 the option “I was forced to enroll in this workshop”, the remaining ratings being 0.

Table 12: Motivation (KEK participants)

Motivation (from 0=inessential to 4=very important) I am interested in learning design I need to learn about learning design People I work with believe learning design is important

I was forced to enrol in this workshop

mean 3,73 3,27 3,67 0,33

st. dev. 0,46 0,88 0,62 1,05

Participant’s opinions about the workshop (Levels 1, 2 and 3 of Guskey’s model).

Looking at the evaluation provided about the workshop, and in particular at participants’ reactions (Level 1), it seems that the most appreciated elements of the workshops at KEK were the relevance of the contents and the quality of presentations. Similarly to what happened at the OU, hands-on activities and quality of discussion were also positively

(29)

29 evaluated. In contrast, the suitability of the workshop with regard to prior competence was judged just adequate and timing was very badly rated. In the free comments, most of the respondents claim that time devoted to familiarization with, and presentation of, the ILDE was definitely insufficient (Table 13).

Table 13: Reactions (KEK participants)

R e ac tion s (f ro m 0 =v ery lo w t o 4= ve ry h igh) Su ita b ili ty with re gard t o y o u r p rior comp eten ce Re le van ce o f t h e w o rk sh o p co n ten ts Qu alit y o f p re se n ta tio n s Qu alit y o f t h e d is cu ss ion Ef fe ctiv en es s o f h an d s-o n act iv itie s Ad eq u acy o f t im e sch ed u le Ad eq u acy o f r o o m s an d facilitie s mean 2,50 3,33 3,33 3,20 3,20 1,47 3,20 st. dev. 0,71 0,49 0,62 0,56 0,86 1,19 0,56

As far as the evaluation data provided by participants regarding their learning at the

workshop (Level 2), ratings are quite high, and participants are positive about the possibility to use the ILDE and apply what they have just learnt in their daily design practice.

Table 14: Learning (KEK participants)

Learn-ing (from 0=very low to 4=very high) I have learnt a lot from this workshop I look forward to putting into practice what I have just learnt about learning design I intend to use the ILDE in my future learning design activities Applying what I have learnt during this workshop will improve the quality of my courses What I have learnt in this workshop will speed up my learning design activities mean 2,93 2,87 3,00 3,00 2,93 st. dev. 0,80 0,92 0,76 0,76 0,80

Participants declare they have created (i.e. conceptualized or authored) a mean of 1 new learning design each, using the ILDE, while no learning designs were implemented per participant, because the implementation phase at KEK was only demonstrated (and not directly acted by participants, see D4.1, Rudman & Conole 2014).

Then participants were asked to evaluate their most complete learning designs and the ratings they provided are contained in Table 15. These ratings are not particularly high, probably because these designs represent the participants’ first attempts at creating a design in the ILDE.

(30)

30 Table 15: Learning: ratings of the designs (KEK participants)

Learning

Completeness Originality Complexity Reusability

mean 2,73 2,50 2,36 3,27

st. dev. 0,47 0,85 0,50 0,79

As to the Level 3 (Organization support and change), people are pretty confident that what they have learnt (in terms of methods) could be transferred to their colleagues, but not so much that they would adopt the ILDE. This may derive from the fact that in Greece infrastructures and teacher technological skills are on average insufficient, as the teachers explain in the open question.

Table 16: Organizational support and change (KEK participants)

Org. support & change

What I have just learnt about learning design could alter the existing practices in my institution

The adoption of the ILDE as institutional tool to support learning design is likely to be fostered by my institution

What I have just learnt about learning design could be transferred to my colleagues ILDE could be adopted by my colleagues mean 2,79 2,43 3,21 1,93 st. dev. 0,70 1,02 0,70 1,07

Lastly, when asked to list three words to describe the workshop, participants expressed the following (see Figure 6). As one can see from the Worlde, ‘interesting’,

‘collaboration/collaborative’ and ‘experiential’ are the most recurrent terms, but similarly to what was declared at the OU, also here teachers said the workshop was ‘short’.

(31)

31 The most appreciated activities were those carried out in groups (because of the possibility to exchange views with others) and the practical and experiential ones; besides, the presentation of collaborative strategies and techniques was also appreciated.

All the suggestions for possible improvements concerned a longer duration of the event and a better timing.

Participant’s opinions about the ILDE

As far as the evaluation of the ILDE, all the participants positively rated both the ease of use and usefulness of accessing the various tools within the platform and found it pretty easy to understand the differences among them.

Table 17: Functions to create new designs (KEK participants)

Functions to create new LdS

how many say they did it

ease of use (mean /st. dev.)

usefulness (mean / st. dev.) understanding that the ILDE

provides access to a variety of tools

16 3,13

0,52

3,19 0,54 understanding the differences

between these tools

16 3,00

0,76

3,06 0,77 finding these tools in the ILDE 0 - - access/enter these tools from

the ILDE interface

16 3,00

0,53

3,06 0,57

As a matter of fact, though, according to the data tracked by the ILDE itself, only a subset of the Conceptualize functions were explored during the workshop, namely Course Map, Persona Card and the Image upload function (see column ‘tracked data’ in Table 18). The evaluation of the ease of use and usefulness of Course Map and Persona Card is positive but not outstanding. Besides, at the KEK workshop, a customized version of the Design Narrative tool was used (paper-based), and this is what the participants rated under “For other

conceptualizations”. Given that this paper-based tool was judged so positively in this context as far as both its ease of use, as well as its usefulness, it was agreed that this tool will be integrated into the ILDE for the next round of workshops.

Table 18: Conceptualize functions (KEK participants)

Conceptualize functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) Course Map 16 2,75 0,68 2,75 0,68 3

(32)

32 Conceptualize functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) Design Pattern 0 - - 0 Design Narrative 0 - - 0 Persona Card 16 2,53 0,83 2,63 0,89 7 Factors and Concerns 0 - - 0 Heuristic Evaluation 0 - - 0 CompendiumLD (upload) 0 - - 0 Image (upload) 0 - - ₇ For other conceptualizations 16 3,25 0,68 3,27 0,70 N.A.

Among the Author functions, participants used WebCollage (11 designs were created with WebCollage at the workshop, see last column of Table 19), and opinions about it are positive, both as far as ease of use and usefulness are concerned.

Table 19: Author functions (KEK participants)

Author functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) WebCollage 16 3,31 0,70 3,31 0,70 11 OpenGLM 0 - - N.A. CADMOS 0 - - N.A.

Unfortunately, no answers have been provided as far as the Implementation functions of the ILDE, even if some activities were registered by the system (see column ‘tracked data’ in Table 20).

Table 20: Implement functions (KEK participants)

Implement functions (from 0=min. to 4=max.) how many say they used it ease of use (mean /st. dev.) usefulness (mean / st. dev.) n.° of designs (tracked data) Implement in a VLE through Glue!-PS 0 - - 3

See your VLE 0 - - N.A.