Structure of methodology chapter

The changes, improvement or deterioration, identified in the argumentative essays between the baseline and argument diagramming phase are investigated because they are important point of reference for the other changes, i.e. the change in process and the change in awareness. Section 4.3 describes the essay analysis methodology which investigates changes in written argumentation strategies. In order to avoid bias, the analysis of essays, undertaken by the writer of the thesis, was carried out blindly to the participant’s identity, baseline or argument diagramming phase and paper or computer condition.

Section 4.4 describes the analysis methodology of investigating changes in planning and writing process. The process investigation is based on analysis of think aloud talk, video transcripts capturing planning and writing activities, and planning products, such as notes and

diagrams, produced at both phases. A comparison between the baseline and the argument diagramming phase is conducted.

The method for collecting the interview data is presented in Section 4.2.10 while the method of interview analysis is presented in chapter 6.

Section 4.2 presents data collection instruments and methods and issues related to the design of the study. Section 4.3 presents in detail the method of analysis of essays and Section 4.4 the analysis of process data.

4.2 Design of Study 2

4.2.1 Participant recruitment

An electronic invitation to participate in a study on essay writing for English native speakers was sent out to all first-year undergraduates of the University of Birmingham during the latter part of the academic year (March 2010). The invitation informed the students about the nature of the study and the compensation (30 GPB) they would receive once they complete a 7 hour- long study over 3 days.

Out of the 32 who responded to the e-mail, the first 16 students were involved in the pilot phase, and the 16 who responded, were scheduled to participate in the main study). The latter 16 participants, 5 males and 11 females, were randomized to 4 groups (computer group essay topic A, computer group essay topic B, paper group essay topic A and paper group essay topic B) using a web based random assignment tool1 (Table 4.1).

The ideas and beliefs that people hold about knowledge and knowing, in other words their epistemological beliefs, are influenced by their academic experiences and may also affect

1

their critical thinking skills (Hofer, 2001). Although many studies refer to the value of generic skills in argumentation, argumentation differs across disciplines (Andrews, 2010). As shown in Table 4.1, the 16 participants came from a variety of disciplines and this is possible to affect the approach they took when they engaged in argumentation.

4.2.2 Argument diagramming editor

The software package Rationale 2 TM was used2 to support the computer-based practice in argument diagramming (Figure 4.1). The argument diagram editor was chosen for three reasons: First, it supports construction of argument diagrams that resemble those of the dialectic method used in study 1a and 1b. Second, the diagram notation of Rationale, and its predecessor Reason!Able, was designed to support the development of critical thinking skills.

2

http://rationale.austhink.com/

It is the outcome of many years of research at University of Melbourne and commercial development. It draws on the analytic models of argument but also integrates the dialectic dimension. Third, Rationale is commercial software which has already been used in a number of published studies (Carrington et al., 2011; Rider & Thomason, 2008; van Gelder, 2003) to support the development of critical thinking skills. As such there are limited usability issues.

4.2.3 Task

Each participant completed the study independently and received individual instruction by the experimenter. The study consisted of two writing sessions, writing task I (baseline), writing task II (argument diagramming), and a training session in argument diagramming in between (Table 4.2). After each writing session the participants were interviewed. Before starting the writing task the participants were asked to practice thinking aloud while drawing the blueprint of their house on paper and then to compose a paragraph about whether smoking should be banned in public. Twenty minutes were allocated for practicing thinking aloud.

Baseline: All participants were asked to write their first essay (baseline) following their usual planning strategies, and produce a text of up to 1300 words in 2 hours, including time for planning or drafting.

Training: All participants were individually introduced to using either the paper-based or the computer-based planning method, according to group allocation, with demonstration and hands on exercises (1 hour). Then they were given one more hour to practice the diagram method. They worked on building an argument diagram on the same topic they write the baseline essay on and they also reported how they would linearize the diagram in text

P ar ti ci p an

ts Name3 Age School Baseline

essay topic

condition

P1 Fern 19 Political science and international studies

A PC

P2 Billy 18 Mathematics A PC

P3 Rea 19 Sociology A PC

P4 Shaun 21 English Literature and Philosophy A PC P5 Harriet 18 School of Geography, Earth and

Environmental Sciences

B PC

P6 Ann 18 Biological Sciences B PC

P7 Mary 19 Classics, Ancient History and Archaeology

B PC

P8 Charlie 19 Electrical and Electronic Engineering B PC

P9 Harry 20 Economics A Paper

P10 Fiona 19 Political science and international studies

A Paper

P11 Sheila 19 English Literature and Philosophy A Paper

P12 Diane 29 Social sciences A Paper

P13 Pandora 19 Biosciences B Paper

P14 Liana 19 Biosciences B Paper

P15 Deana 19 Political science and international studies

B Paper

P16 Anthony 19 Mathematics B Paper

Table 4.1: List of participants and group allocation

3

.

Table 4.2: Design of Study 2

Baseline n=16 Practicing in applying thinking aloud protocol 20-30 minutes

The participants are asked to think aloud while drawing the blueprint of their house or flat and writing a paragraph on what they would do if they had 10.000 pounds

Writing Task I (1300 words)

2 Hrs Essay topic A or B is randomly assigned. No advice on planning is instructed.

Interview 20 minutes

The participants are interviewed regarding their difficulties with argumentative writing and usual and current process in formulating

argumentation structure while planning or composing. Training in argument diagramming : Random allocation to Computer group (n=8) or Paper group (n=8) Introduction to using the method (either on paper or on computer) 1 hour approx.

Similar steps and identical examples are used in training the paper and computer group. The introduction is given individually to each subject by the experimenter.

Practice on building a diagram

1 hour The participants are requested to draw an argument map on the topic of Writing Task 1.

Argument diagramming Computer group (n=8) or Paper group (n=8) Writing Task II (1300 words)

2 Hrs The second topic, A or B is assigned. The participants are advised to use the diagram method, on paper or on computer, depending on their allocated group.

Interview 30-40 minutes

The interview focus on impressions about the planning method, gained knowledge about planning and linearizing strategies, and change in attitude regarding difficulties with

argumentative writing. Total 7 hours approximately

Argument diagramming: The participants were asked to write an essay of similar length on the second topic and use the diagram method, either on paper or on computer, depending on the group they were allocated to. They had 2 hours to complete.

Interview: All participants were interviewed twice. First, right after they wrote the baseline argumentative essay and secondly after they finished writing the second essay. The semi- structured interview lasted on average 20 minutes after the baseline essay, and 30 minutes after the posttest essay.

4.2.4 Essay topics

The topics are believed to raise issues that most participants are familiar with (Table 4.3). The participants were not given access to relevant content, e.g. articles, before composing the essays.

All participants wrote an essay on each of the following topics A and B. Following a random allocation 8 participants were assigned to write on topic A and 8 on topic B. The topics were then reversed for the posttest essay (See Table 4.1 for allocation of participants to topics).

Topic A:

Nowadays people use more and more the internet and specifically what is called social networking tools in their social life. Websites, like the Facebook, reach record high visits every day. Should people use the internet to build relationships or not? Where do you stand in this debate?

Topic B:

There is an on-going debate about whether undergraduate students should pay tuition fees. Should students in higher education be charged tuition fees or not? Where do you stand in this debate?

4.2.5 Training in argument diagramming

The finalisation and testing of material used during the introduction of participants to the diagramming method and practice (training session), took place during the pilot studies. After many iterations and revisions it has been decided that the participants should be gradually introduced to the use of the diagrammatic method, following the same scripted steps, on paper or on computer group, with the guidance of the experimenter and hands-on exercises and practice (the steps and material used during the computer-based training are included in Appendix VI, p.428). For example, after being introduced to the basics of the diagram notation the participants were asked to assemble a diagram on the screen, thus they become familiar with both the software and the notation. A similar exercise was designed for the paper group, providing paper cuts and markers. In the end they were asked to produce a diagram plan from scratch relying on the content of the baseline essay.

4.2.6 Experimental set up

While each participant was left to work alone, the experimenter was sitting in the same room behind a panel (Figure 4.2 and Figure 4.2). The experimenter could observe the process of the participants on screen, respond to participants’ questions but also supervise the recording of the process, making sure that online data were captured on video from various sources. Observing the process in real time allowed the observer to gain an overall idea about the process of the participant which informed the discussion during the interview.

Figure 4.2: Experimenter’s position during data capture

4.2.7 Think aloud talk as process data

Think aloud data, known to provide insight in the process of planning and writing (Smagorinsky, 1994), were collected at two instances, while the writers wrote the baseline essay and the posttest essay. Think aloud protocols are considered a valuable method for collecting process data (Kuusela & Paul, 2000; Ransdell, 1995) despite criticisms about disrupting the process of writing (Janssen, van Waes, & van den Bergh, 1996).

4.2.8 The 6 point criteria list

In order to collect online data that could inform the formulation of argument structure, the participants were given a 6-point list referring to aspects of argumentation structure, and were asked to tick next to each item whenever they consider it, during planning and writing (Table 4.4). The list is suggesting the principles of integration of arguments and counterarguments, of formulating and supporting a position as well as refuting counterarguments.

The aim of using this list was twofold: First, it allows capturing data about when the participants were actual ticking off an item, how often, and after what sequence of actions. This can enhance the data gathered from think aloud protocols. Second, the list introduced a set of specific goals for the writing task defining what is expected regarding the argumentation structure. Specific goal instruction, such as to generate counterarguments, is found to have positive impact on writing, in comparison to general goals such as to persuade the reader (Ferretti et al., 2000; Nussbaum & Kardash, 2005). However, the aim of the list is not to test the effect of instructing goals. The list communicates to participants the requirements of the writing genre in which they were expected to compose, thus unifying the representations of the task amongst the participants, who were coming from difference course and schools of the university. Third, as it became evident during the pilot discussions,

reference to the ticked or non-ticked items enriched the discussion about the difficulties and strengths of the participants during composing (in the interview the participants were asked to report which item they found more difficult and easy to tick).

Table 4.4: Criteria list

Criteria of good argumentation √ √ √ √ √

Clear position. Do I take a clear stand in the debate?

Supporting reasons. Do I provide reasons to support my position?

Counter argumentation (objections). Have I presented objections or counterarguments?

Refutation (rebuttals). In view of counterarguments, that may weaken the strength of my position, have I tried to refute them, that is, to say they are wrong?

Clear argumentation. Are the statements I present, the supporting reasons, the counterarguments and the refutations, clear enough? If not, have I provided further statements that clarify, illustrate or even enhance them?

Final conclusion. On formulating my conclusion in this debate, do I take into consideration the supporting reasons, the counter argumentation and refutation?

4.2.9 Capturing process data on video

Many technical issues regarding the capturing and synchronising of video streams were resolved during the pilot tests. Video data were captured from three aspects: a) a bird’s eye view, capturing gestures and activity on paper, such as when the participant is planning the essay on paper during the baseline phase (Figure 4.4 A) or doing a diagram plan on paper during the argument diagramming phase (Figure 4.4, B) b) a screen capture view, where we can follow the participant’s writing process on the computer screen and c) the face view, capturing the face and gestures of participants and the moments they turned to the screen or the paper sheets. The three views were later synched in one video stream to facilitate analysis (Figure 4.4 C and D).

A _B

C D

4.2.10 Interviews

A semi-structured interview was conducted immediately after completing the baseline and posttest essays with all participants. The structure of the baseline and the argument diagramming interviews was similar, although the argument diagramming interview included more questions and tended to last longer (See Appendix IV and Appendix V).

In the baseline interview, after a couple of warming up questions, such as what the participant thought about the topic and whether argumentative essays were assigned during their first year at university, the first part of the interview focused on the participant’s attitude towards argumentative writing, the perceived difficulties and strengths, and the remarks the participants usually receive regarding their argumentative essays. In the second part, the questions focused on planning strategies. In particular, each participant was requested to give an account of the process he or she followed while planning the essay. Segments of the participant’s captured video were used as a stimuli of recall during this part of the interview (Gass & Mackey, 2000), as well as the plans. Additionally, the participant was offered the opportunity to use segments of the captured video, or the produced plan and text, to clarify or elaborate his or her account of the process. The participant was then asked whether the given account was representative of the usual planning strategy and if not was asked to elaborate. Then, in order to shift the focus to argumentation, the interviewer would drew the attention to the item that was least ticked and mostly ticked on the list, asking from the participant to name the item of the list he or she find most difficult to tick, and why. In the third part of the interview, each participant was questioned about their linearization process, how the plan helped to write the essay and how they ordered and organised the content of the plan in the essay. As in the previous part, they were asked again which point of the item they find more difficult to tick during writing the essay.

The argument diagramming interview followed similar structure and questions adding a few further questions about the diagramming method. In the first part of the argument diagramming interview, there were a few general questions regarding the topic, and a request to give a first impression about the diagramming method. Often the latter was enough to start a vivid discussion. In the second and third part, the participants were asked about how they used the diagram to plan and then how the diagram helped them to write the essay. Similarly to the baseline interview, reference to the item that was least and mostly ticked was made, and the participants were encouraged to discuss. The fourth part of the interview was quite important as it elicited the participants view about the advantages and disadvantages of the diagramming method. Crucially, the participants were asked to compare their process of essay writing before and after using the diagramming method. Furthermore, they were asked to define if and what they learned from using the diagramming method. Finally they were asked to recall what they had previously, in the baseline interview, mentioned as a difficulty and discuss if and how the method helped them to improve.