Assumptions underlying the qualitative study

As outlined in chapter 3.4, successful usage of the CING is likely to depend on learners’ application of steps that are reflected in the models of information specification and autonomous learning. In the following, I discuss how these two models were integrated into the interview sessions.

Information Specification

Schnotz and Zink (1997) investigated knowledge acquisition with hypertext and linear text with and without a predefined learning task. Their theory of knowledge acquisition is called information specification and clearly outlines a set of actions and knowledge items that are important for successful knowledge acquisition, independent of the type of text involved. In the case of the CING, they can be described as follows (adapted from Schnotz & Zink, 1997, p. 97):

1) Adequate specification of information 2) Number of information goals pursued 3) Degree of specification

4) Information search

5) Understanding the topological structure of the CING 6) Knowledge about how to move within this structure 7) Assessment of found information

Evaluation of found information for relevance to the learning goal

Theory of Information Specification operationalized in observation categories

We translated the information specification actions into the following categories and questions for our purposes:

1) Adequate specification of the information (learning) goal - Page usage: relevant or irrelevant pages

- Which relevant pages?

- Do they match the learners’ grammar knowledge gap?

- Pages for both of the grammar topics (present perfect/simple past) or only for one?

- Use of navigational tools in the CING - Navigational path through the CING 3) Assessment of found information

- Page usage: relevant or irrelevant

- Page usage: how long does it take to leave an irrelevant/relevant page?

This operationalization made it possible to collect information on processes that learners would not normally verbalize (e.g., specification of information goal) during their CING work. Thus determining the number of relevant and irrelevant pages learners visited and whether these relate to their stated learning gap can provide information on the “number of information goals pursued.” Learners’ “degree of specification” of their learning goal was observed by investigating the visited pages and relating them to the learners’ stated grammar knowledge gap.

We also observed the navigational paths learners followed in the CING in order to collect information on their understanding of the program’s topological structure (Schnotz & Zink, 1997) as well as on the strategies they used for finding information.

Finally, we investigated how many relevant/irrelevant pages learners visited and how long it took them to leave them. Considerable time spent on irrelevant pages is a clear indication of difficulties in correctly evaluating the relevance of given information for the task at hand.

Theory of Information Specification operationalized in interview questions

The following categories (see also Appendix G2) are based on the model of information specification (see Figure 26) and were established to collect information on particular learning goals and the reasons for learners’ use of specific pages and navigational tools.

Degree of Specification

1. Question 4/3a): Erkläre mir nochmals in Deinen eigenen Worten, was Deine Aufgabe in der CING war. Was solltest Du tun? (Please explain in your own words what your task in the CING was. What were you supposed to do?)

2. Question 4/4a): Kannst Du Dich erinnern und mir genau erklären, was Du in der CING als erstes getan hast, um die Aufgabe, die ich Dir gab, zu erfüllen? (Can you remember and explain to me in detail what you did first in the CING today to complete the task I gave you?)

3. Question 4/4b) follows up on the answer to question 4a: Gab es Probleme dabei? (Did you experience any problems performing this first step?

It is crucial for learners to understand the goal of the task that they were given in order to be able to identify relevant actions and materials. Question 4/3a was intended to make learners verbalize this understanding. Questions 4/4a and b aimed at determining whether learners’ first steps in the CING show a form of specification of either information or learning activity. Question 4b was included to find out whether the CING contains obstacles that hinder the application of actions or use of material specified by learners.

Clear Idea of Existing Knowledge Gaps

6) Question 3/7: Zeige mir mit Hilfe dieser Skala, wie gut Deiner Meinung nach Dein Wissen über das Simple Past und Present Perfekt (die Anwendung und Regeln) ist.

(Please show me on this scale how good you think your knowledge of the simple past/present perfect is (in terms of usage as well as rules).

7) Question 3/8: In welchen Bereichen des Simple Past und Present Perfect denkst Du, dass Du noch etwas mehr Übung brauchst? (In what areas of the simple past and present perfect do you think you need more practice?)

Questions 3/7 (based on a 5-point Likert scale) and 3/8 were developed in order to investigate our subjects’ awareness of their own knowledge gaps in relation to their task. As described by Schnotz and Zink (1997), the specification of information goals when solving a task can become more efficient if learners are aware of the knowledge gaps they have in the subject area and integrate this knowledge into their information search (and usage) process.

First Navigation Moves and Comprehension of CING Topic Titles (CING Page Nodes)

1) Question 4/5a: Zeige mir die Seite, die Du als erstes in der CING besucht und gelesen hast.

(Please show me the page that you first visited and read in the CING.) If subjects named the menu or the intro page) they were prompted with:

2) Question 4/5b: …und welche Themenseite? (… and what content page?)

3) Question 4/8a: Kannst Du mir Seiten in der CING zeigen, die nicht relevant für Deine Aufgabe waren, die Du aber zuerst für relevant gehalten hast? (Can you show me pages in the CING that were not relevant to your task, but that you originally thought were relevant?)

If learners named such pages:

4) Question 4/8b: Warum sahen sie für Dich relevant aus? (Why did they seem relevant to you?)

5) Question 4/9: Zeige mir auf dieser Skala, wie schwierig es für Dich war, relevante Seiten in der CING zu erkennen. (Please show me on this scale how difficult it was for you to identify relevant pages in the CING.)

Questions 4/5a and b were aimed at understanding whether learners’ comprehension difficulties uncovered in the quantitative analysis were mirrored in their page use. The type and content of the first page learners visited provides information on task realization, the integration of knowledge gaps, and the need for more support on how to use the CING. Questions 4/8a and b explore how well learners can identify relevant pages and navigate to them.

Learners were also asked about the degree of difficulty they experienced in recognizing and finding (relevant) pages (question 4/9). This question was included in order to elicit a statement on learners’ page selection experience for their entire CING session.

Use of Navigational Tools

1) Question 4/11a: Kannst Du mir erklären, wie Du am besten von der Startseite zu den Themenseiten navigierst? (Can you explain to me how to best navigate from the intro page to the topic pages?)

2) Question 4/11b: Und wie navigierst Du am besten von Themenseite zu Themenseite? (And what is the best way to navigate from topic page to topic page?)

Navigation in the CING can take place by following the CING structure beginning with the intro page, skipping levels in the CING structure, or by circumnavigating it entirely (such as by using the Sitemap [Figure 8] and directly clicking on a page deep within the CING

structure). Although learners’ behavior in the CING must not be inefficient per se if their navigation only happens outside the CING structure, this type of navigation hints at potential problems with the CING titles or content structure. Question 4/11a aimed at determining which navigation paths learners followed successfully and which they preferred. We asked subjects to explain these paths as if they were introducing the CING to a learner new to the tool.

Selected Material Relevance (for learning task)

• Question 4/5c: Woran erkennst Du, dass diese Seite relevant für Deine Testvorbereitung ist? (How can you tell whether this page is relevant for preparing for your test?)

• Question 4/8a (See above)

• Question 4/8b (See above)

• Question 4/6a: Welche Themenseiten haben Dir am besten bei der Vorbereitung auf die Aufgabe geholfen? (Which topic pages were most helpful to you as you prepared for the test?)

• Question 4/6b: Welche Art von Material war für Dich am hilfreichsten? Warum? (What type of material was most helpful to you? Why?)

The ability to understand the task and navigate efficiently in the CING material structure is of little use if learners are unable to assess the relevance of the material they find. Question 4/5c was meant to investigate learners’ strategies for assessing the relevance of material, while questions 4/8a and b were intended to collect information on learners’ ability to recognize relevant and irrelevant pages in the CING. Questions 4/6a and b address learners’ ability to assess the material’s relevance for their own learning strategies.

Autonomous Learning

The quantitative data analysis (see “Empirical analysis” section) showed that the subject group had only little or no experience with autonomous language learning and that the majority of subjects was used to teacher-guided language instruction. In order to investigate which strategies of autonomous learning our subjects were able to apply despite their lack of experience, I created a model of autonomous learning in the CING, adapted from a model by

Ziegler et al. (2003) and other researchers in the field (Holec, 1981; Boekarts, Pintrich, & Zeitner, 2000; Benson, 2001) (see Figure 28).

This combined model of actions in autonomous learning (based on Ziegler et al. 2003, Holec, 1981; Boekarts, Pintrich & Zeitner, 2000; Benson, 2001) includes the following steps:

− Understanding of the task and setting learning goals accordingly (task realization)

− Selection of relevant materials

− Learning strategy selection and application

− Monitoring success of learning strategy application

− Adjustment of strategies if necessary

− Evaluation of the entire approach with the help of feedback

Given the similarities between autonomous learning (AL) and information specification (IS) (see chapter 3, “Theories of learning with hypertext” and “Theories of autonomous learning”) (see Figure 28), all interview questions regarding information specification discussed above also apply to autonomous learning. Despite this overlap, each theory provides a different degree of detail on the actions in question. While IS only generally specifies the processes involved in the action of learning (“semantic processing of information”), the methodology of AL includes as many as three actions to specify learning processes. On the other hand, the action of “selection of relevant information” (AL) involves two actions (“information search” and “assessment of found information”) in IS.

In addition, I developed four questions to operationalize the three AL actions of “selection and application of relevant learning strategies,” “assessment of learning strategies (if necessary: adaptation of strategies),” and “assessment of success of learning strategies and learning approach” (Figure 28).

Theory of autonomous learning operationalized in interview questions

Learners’ existing learning strategies

Question 4/3b: Welche Lernaktivitäten helfen Dir normalerweise (ohne CING) bei der Lösung einer solchen Aufgabe? (What learning activities usually [without the CING] help you solve a task like this?)

This question aimed at determining whether the CING supports or hinders learning strategies hypertext novices bring to a web-based, autonomous learning tool.

Learning Strategy Application

1) Question 4/7: Was hast du auf den ausgewählten Seiten getan für deine Testvorbereitung? (What did you do on the pages you selected in order to prepare for the test?)

The learner log files on CING behavior (see “Background to evaluation” section) provided no information on users’ learning behavior on the content pages they visited. Question 4/7 fills this gap by investigating what learners actually did in the CING to improve their knowledge of the simple past and present perfect.

Assessment of learning strategy and learning approach: Problem-solving strategies

− Questions 4/10a and b: Bereitete Dir die CING Probleme während deiner Aufgabenbearbeitung bzw. Testvorbereitung? Welcher Art? (Did the CING cause you any problems during your task work/ test preparation? What kind of problems?)

− Question 4/10c: Konntest du diese Probleme lösen? (Were you able to solve these problems?)

As the CING does not provide support if difficulties in learning with the tool arise, users’ own problem-solving strategies are a vital part of a successful learning experience with the CING. Question 4/10a focuses on learners’ CING problems; question 4/10b on their abilities to solve these problems.

Clarification of quantitative study results

The following interview questions were intended to clarify inconclusive results from the quantitative data analysis.

Prior experience with the CING

Question 4/1: Du hast schon ein bisschen Erfahrung mit der CING: War die heutige Arbeit mit dem Programm einfacher für Dich als im Dezember 2005? (You already have some experience with the CING. Was working with the program today easier for you than it was in December 2005?)

This question directly relates to my seventh research assumption (see chapter 3, “Theoretical aspects of second language acquisition”): Learning with hypertext differs considerably from traditional learning. Goal-oriented navigation and orientation in the hypertext structure can require prior experience with hypertext material. Now that learners had used the CING at least once in the first research session in December 2005, question 4/1 could inquire into whether their experience with the CING had been conducive to their work during the second session.

Subjects’ problems with autonomous learning in the CING

Question 4/2: Du musstest die CING heute und auch im Dezember ohne Lehrer oder Instruktor nutzen. Bereitete Dir das Probleme? (Today and in December you had to use the CING without a teacher or instructor. Did this cause any problems for you?)

This question directly relates to my eighth research assumption: Autonomous learning strategies help learners to define learning goals, select relevant learning materials, and apply appropriate learning steps and strategies for their task. Question 4/2 was included to collect learners’ statements regarding difficulties the quantitative data indicated they encountered while working autonomously in the CING.

Learner usage of the simple past content pages

Question 3/7b: In welchen Bereichen des SP/ PP denkst du, dass Du noch etwas mehr Übung brauchst? (In what aspects of the simple past and present perfect do you think you need more practice?)

Question 3/4a: Aus deiner Erinnerung an die Forschungssitzung im Dezember letzten Jahres, was war für Dich das schwierigste in der Sitzung? (What do you remember being most difficult about the research session in December of last year?)

Questions 4/4a and b: Kannst Du Dich erinnern und mir genau erklären, was Du in der CING als erstes getan hast, um die Aufgabe, die ich Dir gab, zu erfüllen? (Can you remember and explain to me in detail what you did first in the CING today to complete the task I gave you?); Gab es Probleme dabei? (Did you experience any problems performing this first step?)

Questions 4/5a and b: Zeige mir die Seite, die Du als erstes in der CING besucht und gelesen hast! Welche Themenseite hast Du zu allererst besucht? Warum? (Please show me the page that you first visited and read in the CING. What content page did you first visit? Why?)

Question 4/8a and b: Kannst Du mir Seiten in der CING zeigen, die nicht relevant für Deine Aufgabe waren, die Du aber zuerst für relevant gehalten hast? (Can you show me pages in the CING that were not relevant to your task, but that you originally thought were irrelevant?); Warum sahen sie für dich relevant aus? (Why did they seem relevant to you?) Questions 4/10a and b: Bereitete Dir die CING Probleme während deiner Aufgabenbearbeitung bzw. Testvorbereitung? Welcher Art? (Did the CING cause you any problems during your task work/ test preparation? What kind of problems?)

Question 4/11c: Gab es Probleme beim Navigieren in der CING? Welche? (Did you have problems navigating in the CING? What kinds of problems?)

Question 4/12: Welche zusätzlichen Informationen in der CING hättest Du Dir für die erste Arbeit (Dez05) mit der CING gewünscht, damit diese Dir leichter gefallen wären? (What additional information did you wish the CING had provided that would have made it easier to work with the first time [in December 2005]?)

Question 4/13: Gibt es etwas, das Dir in dieser Sitzung an der CING aufgefallen ist, du aber noch nicht erwähnt hast? (auch wenn es Dir noch so unwichtig erscheint) (Is there anything that you noticed about the CING in this session that you have not yet mentioned, no matter how unimportant?)

The observational data on learners’ behavior in the CING (December 2005) showed that many avoided all content pages on the simple past . Questions 3/7b and 4/5a and b were included to find out why this was so. Questions 4/5a and b were meant to help me understand

what influenced learners’ first steps in their material search in the CING. I expected the reasons for learners’ page selection to be manifold, ranging from their awareness of grammar knowledge gaps (questions 3/7 and 8), to the content page’s position within the content structure/index (questions 4/8a and b), to learners’ experience with the CING (questions 3/4, 4/10a and b, 4/12, and 4/13), their path in the tool (questions 4/4b and 4/11c), as well as their recognition of relevant content page titles (question 4/8b). Question 4/11c was also intended to lead to a better understanding of the impact of CING titles on learners’ CING behavior.

Reasons for negative experiences with CING Feedback

Question 4/2: Du musstest die CING heute und auch im Dezember ohne Lehrer oder Instruktor nutzen. Bereitete Dir das Probleme? (Today and in December you had to use the CING without a teacher or instructor. Did this cause any problems for you?) Questions 4/10a and b: Bereitete Dir die CING Probleme während deiner Aufgabenbearbeitung/ bzw. Testvorbereitung? Welcher Art? (Did the CING cause you any problems during your task work/ test preparation? What kind of problems?)

The quantitative data analysis on learners’ experience with the CING showed that about a third of all learners had problems with the CING feedback function. Questions 4/2 and 4/10a and b as well as the observation data in the interview session were intended to help provide an explanation for this finding. Instead of asking learners specifically about their problems with the feedback, we posed two general questions on learners’ problems with the CING. If the feedback had been a serious problem for learners, we expected this to come up in their responses to these general questions. The general nature of the questions also gave us the opportunity to pose more focused additional questions to clarify learner answers, such as regarding the negative relation the quantitative data revealed between learners’ CING feedback experience and their CING title comprehension.

Reasons for problems with the CING material’s salience

Questions 4/10a and b: Bereitete Dir die CING Probleme während deiner Aufgabenbearbeitung bzw. Testvorbereitung? Welcher Art? (Did the CING cause you any problems during your task work/ test preparation? What kind of problems?)

Question 4/12: Welche zusätzlichen Informationen in der CING hättest Du Dir für die erste Arbeit (Dez05) mit der CING gewünscht, damit diese Dir leichter gefallen wären? (What additional information did you wish the CING had provided that would

have made it easier to work with the first time [in December 2005]?) Question 4/13: Gibt es etwas, das Dir in dieser Sitzung an der CING aufgefallen ist, du aber noch nicht erwähnt hast? (Auch, wenn es Dir noch so unwichtig erscheint.) [Is there anything that you noticed about the CING in this session that you have not yet mentioned, no matter how unimportant?].

The quantitative data analysis indicated that learners found the CING material to not always be salient. We decided against asking learners directly about “salience”, as the term and concept itself might not be understandable to them. Instead, the general questions (4/10a and b, 4/12 and 4/13) on material comprehension provided information on this issue. Again, if a