Exploratory Study

The first study [175] focused on the research question how different tools used in learning processes affect the way learners execute their learning processes using web resources. Further, the question how explicit prompts can be given in order to initiate goal–setting, planning and reflection processes [21] was targeted. 64 participants (all psychology bachelor students in their first two semesters between the age of 19 and 28 years,76.6% being female due to the field of study) were asked to take part in a study that targeted learning using the German Wikipedia for 45 minutes. Four different treatment groups were formed by random assignment:

• Control Group 1 (CG₁1, n= 16) used only the tools the web browser Firefox provides. Participants were allowed to bookmark found resources.

• Control Group 2 (CG12, n = 18) used pen and paper as the means to persist their findings. The

participants of this group were allowed to take notes during the search.

• Treatment Group 1 (TG11, n= 15) used ELWMS.KOM without specific instructions how to proceed.

Thus, this group represents a realization of indirect scaffolds as defined in section 6.2.

• Treatment Group 2 (TG₁2, n = 15) used ELWMS.KOM and were asked to execute metacognitive processes by metacognitive prompts in the learning phase. Before learning, participants of this group were prompted to set their search goals. After the learning phase, the participants were instructed to reflect over the found resources and their created goals for five minutes.

Study Design

The study followed a design that is briefly outlined in the following: Before the participants started with the learning episode, they were given a pre–test questionnaire that collected basic demographic data, computer literacy (based on their self–conceptualization), learning competencies (i.e. the competencies 6 _{http://de.wikipedia.org/Portal:Antike, retrieved 2010-12-01}

7 _{Free Version of Camtasia Studio 3.1.2, http://www.techsmith.com/camtasia/, retrieved 2006-06-15}

to plan and structure their learning processes based on items from [195]) and their current motivation and confidence in their learning competencies. Further, their emotional traits were measured according to PANAS [111], a standardized questionnaire aiming at measuring positive and negative emotions. Then, depending on their experimental conditions, they were given a five–minute introduction in either ELWMS.KOM or Firefox. Before learning, all participants processed a performance test containing 30 multiple–choice questions about the topic Classical Antiquity. They were suggested that the same test would be given again in a post–test, enabling competent learners to identify knowledge gaps in the pre–test and reformulate these into learning goals. In the learning phase, all groups were given hints about the remaining learning time after 25 and 40 minutes. After learning, the performance test was handed out again, followed by a questionnaire asking about the participants’ approach during learning, the current motivation and emotions again.

Selected Results

In total, during the learning phase all 64 participants viewed 242 unique web resources, that is on average 17.23 resources per participant (when counting repeated viewing, 22.38 web resources were opened). An important observation was that a majority of goals was rather formulated as a topic than as real goals (e.g. “I want to get an overview of the topic Antiquity”). On inquiry, participants stated that this was due to the given task and time constraints and that the user interface of ELWMS.KOM is only displaying the first words of long goal names.

Selected Group differences

The groups were contrasted over the whole learning process based on different variables gained by analysing the log files and questionnaires. Two group comparisons were executed by applying Student’s t–test [69], for the comparison of more than two groups, one way ANOVAs [69] (Analysis of Variance be- tween groups, comparing group means with each other) were used. Table 6.2 shows selected significant differences between the groups. All significance values in the following have been computed one–tailed.

Variable Contrasted Groups Student’s t–test

Restructure Goalα TG12< TG11 MT G₁1= 7.2; MT G₁2= 3.87; p < .05∗ Revisited Resourcesα TG12> TG11 MT G11= 0.2; MT G12= 1.0; p < .05

∗

Variable Contrasted Groups ANOVA

Opened resourcesα TG11∩ TG12> CG11∩ CG12 F(3/60) = 3.65, p < .05∗ Opened imagesα TG₁1∩ TG12> CG11∩ CG12 F(3/60) = 1.71, p < .05∗ PANAS “active”β TG11∩ TG12> CG11∩ CG12 F(3/60) = 3.19, p < .05∗ PANAS “determined”β TG11∩ TG12> CG11∩ CG12 F(3/60) = 4.60, p = .01∗∗

Table 6.2:1st _{Study: Selected Group differences based on log files and questionnaires. F=F–value, p =niveau of}

significance, M=mean value.∗denotes significance (p< .05),∗∗denotes strong significance (p< .01). Variables marked withαdesignate data obtained from logfiles, variables marked withβ were collected in questionnaires.

Performance was, as expected due to the short learning episode, not significantly different between the groups. However, as TG₁2 “lost” five minutes due to the metacognitive prompts, they had effectively less time for learning. Participants of groups using ELWMS.KOM (TG11 and TG12) browsed significantly

more web resources and opened more images that were partly relevant for understanding the continuity of the learning materials. Further, TG11 and TG12 benefited from using the goal–setting component

emotionally and motivationally: in comparison to the control groups, they felt they executed their search process in a more active and determined way.

The treatment groups differed significantly on how web resources that were already assigned to goals were handled. TG12 changed the already persisted web resources less often, as they already had ade-

quately planned their approach before learning. Thus, they already had a goal to attain and progressed more target–oriented. Further, they revisited their already persisted web resources more often and thus showed to execute reflective processes at the end of the learning phase.

Selected Correlations

Participants that were initially motivated showed a higher willingness to use the goal–setting com- ponent intensively and to delve into the learning process. This can be seen in the correlation of the self–declared motivation in the pre–test and the number of followed image links, the number of set goals and their editing and usage of the offered representation of the goal structure as a knowledge network (for correlation values, see table 6.3).

Variable 1 Variable 2 Correlation r

Opened imagesα Motivation pre–testβ .252∗

Number of set goalsα .221∗

Edit actions on goalsα .226∗

Browsing knowledge netα .259∗

Search literacyβ Opened resourcesα .416∗

Deleted resourcesα .385∗

Computer literacyβ Opened resourcesα .525∗∗

Edited resourcesα .346∗∗

Restructure goalα PANAS mean negativeβ .353∗

PANAS “confused”β .590∗∗

Edited resourcesα PANAS mean positiveβ .326∗

Further usageβ Motivation pre–test / post–testβ .343∗_{, .401}∗

Revisited resourcesα −.375∗

Restructure goalα .343∗

Table 6.3:1st _{Study: Selected significant correlations between variables,}∗_{denotes significance,}∗∗_{denotes strong} significance. Variables marked withαdesignate data gained from logfiles, variables marked withβ were collected in questionnaires.

The higher the self–perceived search and computer literacy, the more learners were able to filter irrel- evant resources and reflect on their already found resources. Additionally, the already saved resources were re–opened and used more often.

If the participants had created a realistic goal structure before having started to learn, they did not have to restructure their goals in the following. At the same time, less negative emotions occurred compared to participants having restructured their goals more often. Especially the feeling of disorienta- tion occurred less often with participants having adequately planned their learning process before–hand. Thus, the planning and pre–arrangement of a realistic goal structure before executing the search us- ing ELWMS.KOM led to a more positive experience of the learning process. Further, the functionality of adapting the resources to the current state of the learning process by stating the relevance of a re-

source and editing the resource’s snippet was perceived as useful. Thus, positive emotions accompany the number of resources that were edited, commented and tagged8.

The less resources the participants collected and the more need they saw to restructure their goal hierarchy, the more the participants stated in the questionnaire that they would like to use ELWMS.KOM or a comparable goal–setting tool in further learning episodes. This may hint that especially learners that do not automatically execute self–regulated processes perceive the need of getting support for self– directed learning using web resources. Furthermore, motivation before and after the learning phase correlates with the desire for using a comparable goal–setting tool in future learning episodes.

Conclusions from Exploratory Study and Implications for Second Study

In conclusion, the presented scaffolds did influence the learning processes, although an impact on the learning performance could not be shown. This was expected due to the short learning episodes, because planning, monitoring and reflecting continuously are expected to have an impact in longer learning settings. Further, in the short time span that the participants learned, the participants could contain all found facts in short–term memory, and therefore the advantage of an elaborate organization of goals and resources was not of primary importance for this learning setting.

Still, other several issues with the study design were encountered. First, the goal was to emulate “real- istic” environments for the learners, i.e. forming a control group learning using bookmark functionality and a pen and paper group. Therefore, the groups were not comparable in all accounts and that might have influenced the learning outcomes. For example, the pen and paper group CG₁2 did not have to learn using a new tool and could quickly outline information and set relations between content that was not possible for the other groups. Additionally, the bookmark group CG11 was lacking the possibility to

save web resource snippets, thus participants had to bookmark the whole page — which many partici- pants thought to be futile, thus not using this functionality at all. Another issue was that the assumption was made that the students had no prior knowledge about the topic Classical Antiquity without proving it. Eventually, the groups using the goal–management component were only briefly trained to using it before learning. This means that computer literacy and experience in using comparable tools had a strong influence on the way students were able to handle ELWMS.KOM.

Thus, the study design and some aspects of the goal–management component were redesigned in order to further improve the evaluative quality of the study.