Research design and methods of the trial

Research design and methods of the trial

This chapter describes the randomised controlled trial, the sampling strategy, the randomisation process, the development of the intervention, the materials used, the training of teachers, and the conduct of the process evaluation.

6.1 The aim of the present trial

To aim of the trial is first to establish whether teaching CT skills to higher education students who are learning English as a foreign language in Lebanon can improve their CT skills. Secondly, it aims to test whether instruction in CT is feasible in a culture where CT is not explicitly promoted. Thirdly, it tests whether explicit teaching using the general approach (i.e. where CT instruction does not require in-depth knowledge of a particular subject or domain) is effective in developing CT skills of higher education students in Lebanon.

The trial therefore aims to answer the following research questions:

RQ 3 Can general CT skills be taught to ELL in higher education in Lebanon within the regular curriculum?

RQ 4a Is it feasible to teach CT skills in an education system which does not generally promote independent thinking and argumentation?

RQ 4b Is it feasible to teach CT skills in a culture where the curriculum is heavily dictated by religion and politics?

RQ 5 Do students' characteristics (e.g. gender, subject major, exposure to a foreign culture, job experience) affect their receptivity to CT skills?

These are causal questions and thus an experimental design in the form of a RCT was deemed the most appropriate for this investigation.

6.2 Rationale for a randomised controlled trial

A RCT involves comparing the outcome of one group which receives the treatment or intervention with a control group. The control group provides a counterfactual by indicating what would have happened to the experimental group had they not received the treatment. The use of a control group to which participants have been properly and

threats to internal validity (Shadish, Cook, & Campbell, 2002; Bryman, 2012). Internal validity refers to the confidence with which experimental research can draw causal conclusions (Bryman, 2012). The validity of a trial largely depends on the ability of the researcher to ascertain that any difference between the two groups is due to the treatment itself and nothing else (Gorard, 2003), hence the use of a control group is necessary to ensure this.

An advantage of a RCT is its ability to counter a variety of threats in the research design (Cook, 2002) as whatever affects the experimental group, other than the treatment itself, is likely to affect the control group too. Random assignment to groups also ensures that there is no selection bias.

While individual randomisation increases the power of detecting effect sizes, cluster randomisation is sometimes necessary in cases where it is not possible to randomise individuals. In this study a cluster RCT was employed where groups by teachers were randomised because the students were allocated by university administration to classes and some teachers teach more than one class (so randomisation by class was not feasible). Cluster randomisation was therefore necessary to prevent contamination (Torgerson & Torgerson, 2001). In cluster randomisation each cluster is considered one case and therefore the more clusters there are the better to be able to detect any noticeable differences between the two groups that are being compared (Torgerson et al., 2010).

6.3 The intervention

The trial was conducted over two terms with two different cohorts of students in the same university. One was conducted in the Spring term and the other in the Fall. The duration of the intervention is one term consisting of seventy sessions in which 11 lessons were used for teaching CT for the treatment group. These lessons were spread over 14 sessions of 50 minutes each. Students in both the control and experimental groups received the same number of sessions. While the control group had lessons based on their regular curriculum in that term, treatment classes had some lessons on CT concepts using materials developed by the researcher which substituted for some of the materials that were part of the regular curriculum and were integrated into the module. Otherwise, course materials and instruction were the same for both groups

This section describes the intervention using the TIDieR (Template for Intervention Description and Replication) checklist (Hoffmann et al., 2014).

Name of intervention

A generic approach to explicit instruction in critical thinking

The rationale for the intervention

The belief is that explicit instruction in CT concepts following the general approach could improve students' CT in the language classroom in higher education. As students in the language classroom must read texts, have discussions, and write essays, there are many opportunities to nurture their CT skills simultaneously with their language skills.

People involved in the intervention

The intervention was designed by the researcher who also provided informal training to the teachers who taught the experimental classes. Participants were students in the English 102 module and their teachers.

The teaching resources used in the intervention

Each CT lesson was prepared on a handout consisting of two to three pages maximum. The answer key to each activity was prepared on a separate document and given to experimental teachers. All the lessons consisted of handouts except for one where students had to watch a short video of around 20 minutes. The video was kept in the language lab which was reserved for the experimental classes to watch. Students were also given a handout to answer some questions while watching and to discuss them afterwards.

Photocopies of the lessons for all experimental students were made available to the teachers a couple of days before the lesson. Teachers received the handouts with the answer key around three days ahead of the time assigned for each lesson.

As students in this trial had no previous exposure to CT, only basic CT concepts were introduced. The lessons revolved around concepts in CT, and the teaching resources were developed by the researcher using and adapting materials from textbooks, newspapers, academic books and websites. Some of the materials were adapted to suit

The lessons that formed part of the intervention and that were given to students as handouts are provided in Appendix 4. The answer key that was given to teachers can also be found in Appendix 4 at the end of each lesson.

The themes covered were a general introduction to logical fallacies, difference between causation and correlation, assumptions and stereotypes, reliability of sources, counter argumentation, in addition to a brief introduction to validity of research and an introduction to biases in surveys and statistics.

It is important to note that focus was on general CT skills and not on the skills that are emphasised in the CCTT, Level Z. The aim of the researcher was not to teach to the test but rather to explore whether CT can be taught.

Lesson One

The first lesson was an introduction to logical fallacies where students were introduced to common fallacies such as hasty generalisation, either-or fallacy and circular reasoning. Students were given a handout with sentences that had weaknesses in reasoning, and then they had to find out and explain why the reasoning in the sentence was flawed. Students were not required to know the names of fallacies. Fifteen sentences with flawed reasoning were included in the handout in the first lesson. Students were first given time to think about the sentences, and then they discussed the possible answers with their class teacher.

Lessons Two and Three

The next two lessons focused on the difference between causation and correlation. Although the difference between causation and correlation was briefly introduced in the first lesson on logical fallacies, a more detailed explanation was provided in these two lessons. In order for students to understand the difference between causation and correlation, students were given two different texts from newspapers forming two different lessons. The first newspaper article (to which two sessions were devoted) was about the link between Facebook and divorce and the second lesson (also done in two sessions) focused on the link between living by the sea and good health. As the focus of English 102 (the regular syllabus) is on writing summaries and paraphrasing parts of a text, students in the experimental group used the two texts on causation versus

Therefore, both groups, the experimental and the control, practised the same skill but the difference was the type of text given.

Lesson Four

Lesson Four was about assumptions and stereotypes. Students in the experimental classes were given a newspaper article about stereotyping female drivers and its negative effect on the driving performance of women. They started the lesson by having a class discussion on what assumptions and stereotypes mean. Just like the previous lesson, they had to summarise the text and paraphrase some parts, skills that have to be introduced and reinforced in the regular syllabus.

Lesson Five

The fifth lesson took a slightly different form from the customary as it involved a TED (Technology Education Design) talk video on assumptions and stereotypes. A Nigerian lady discusses her experience with stereotyping and being stereotyped. Students were given a task sheet with questions so that they stay focused on the video, and then a class discussion followed.

Lessons Six and Seven

Lessons Six and Seven examined the reliability of sources. The aim of the lessons was to make students aware that there is often a hidden agenda behind a claim. They were also taught how to look at the source of information to determine its credibility. Short excerpts taken from various sources were given to students to examine and determine the reliability of their claims. The second lesson on reliability of sources was about KFC. Two short texts, one from a newspaper and one from a website that confirms or debunks questionable stories that go viral on the internet, were given to students to read and examine. Students then had to determine which text was more convincing and they had to provide reasons for their answers in a class discussion.

Lesson Eight

In Lesson Eight, students had to think of counterarguments for short texts. The texts were about vending machines and obesity in children, the usefulness or not of homework, and college attendance policies. Students were asked to think of counterarguments to certain sentences in the texts.

Lessons Nine and Ten focused on studies and surveys. The focus of Lesson Nine was on the validity of a study and the hidden factors that might affect the conclusion in a study. Lesson Ten was on surveys and focused mainly on biases in surveys and in the way questions could be asked.

Lesson Eleven

The last lesson consolidated all the lessons that students had taken. Students were given a text in which the writer defends cheating. Students were asked to identify the writer's argument and the reasons. Then they were asked to examine if there were any logical fallacies in the text. This lesson provided the opportunity for students to evaluate a piece of writing and to engage in critical reading.

Table 6.1 summarises the lessons introduced during the trial and the number of hours spent on each lesson.

Table 6.1 Lessons and number of sessions

Lesson Number of

lessons

Number of sessions

Logical fallacies 1 1

Correlation versus causation 2 4

Stereotypes and assumptions 2 3

Reliability of sources 2 2

Counter argumentation 1 1

An introduction to studies and surveys (linked to reliability of sources)

2 2

Reading critically (a text including logical

fallacies, assumptions, and other related themes)

1 1

Total: 11 lessons

Total: 14 sessions

Procedures used in the intervention

students to voice their opinions as much as possible during the lessons. The number of students in each class ranged between eight to sixteen students. Students would keep the handout with them at the end of the lesson.

Location of the intervention

Classes were held as usual in the classrooms allocated by the Registrar's office. Some classes were next to each other and some others were in different buildings. The university where this intervention took place is a private university in the north of Lebanon.

Duration of the intervention

The intervention first took place in Spring 2014-15 over one term extending from February 4 to May 27 (16 teaching weeks). Then it was repeated with another group of students in Fall 2015-16 over one term extending from September 2 to December 10 (16 teaching weeks). The eleven CT lessons (14 sessions of 50 minutes each) were scheduled to fit in within the term. An average of one or two sessions of CT were given each week (the duration of the term is approximately four months).

Monitoring of the intervention

The intervention was monitored by observing the delivery of the lessons. This was to check for fidelity of implementation (i.e. if the lessons were conducted as planned) and to identify potential barriers to delivery. The researcher was able to observe the lessons of only two experimental teachers each time a new lesson was introduced. As the researcher had her own experimental classes, it was not possible to observe the classes of the other two experimental teachers. Details about the process evaluation are presented in Chapter 10.

6.4 The sample

The trial was conducted with a group of 413 students forming 29 clusters. The trial was first run with a group of 226 university students forming 16 clusters in Spring 2014-15 over one term extending from February 4 to May 27 (16 teaching weeks). The same trial was repeated on another cohort of 187 students forming 13 clusters in Fall 2015-16 over one term extending from September 2 to December 10 (16 teaching weeks). The reason for repeating the trial over two terms was to increase the sample size.

The sample for this study included all students enrolled in the English 102 module. English 102, is a Freshman module - a transitory module between school and university in a private university in Lebanon that follows the American system where the language of instruction is English. Students usually take this module in their first year of undergraduate study or sometimes in their second year depending on their proficiency in the English language. It is a university requirement for students who score between 570 and 597 on the Test of English as a Foreign Language (TOEFL) (paper-based), 440 and 489 on the Scholastic Aptitude Test (SAT), or 7 on the International English Language Testing System (IELTS). Most students in English 102 come straight from school. However, there are some who have come from English 101, which is a lower Freshman module, so this is their second term. Other students include those who have passed the Intensive English module (lower than English 101) and have moved from English 101 to English 102, so this is their second year in the university. To some, English is a second language while to others it is a third language as their second language is French. All students are native speakers of Arabic.

The researcher, who has taught this module for about eight years, has chosen this module to experiment on because a considerable number of students are supposed to have had substantial exposure to English by the time they reach this module. Therefore, students at this level do not have to struggle with the difficulty of learning a completely new language. Students are expected to be proficient in English; however, they do not all have the same level of proficiency in all the language skills. Some students are better writers than others, for example, while others are more fluent in speaking. Classes are somewhat heterogeneous.

Students that formed part of the study are undergraduates from various majors like Engineering, Architecture, Nursing, Biology, Tourism and Hotel Management etc. Students were mostly Lebanese with a few who were born overseas or held foreign citizenship. Students’ ages ranged between 17 and 27 with the majority being 19 years old in both experimental and control.

6.4.1 Description of the regular module lessons (control group activity)

placed on essay writing skills. Students are expected to learn how to write a documented argumentative essay and a research paper using references. The class meets 5 days a week for a 50-minute session every day. Although the module focuses on argumentation and class discussion may be encouraged, not enough time is devoted to teaching CT concepts. Students, for example, are not taught logical fallacies. Nor are they taught to read critically or to question arguments with a critical eye.

The focus of the module is on the writing skill with reading and speaking having a secondary role. The module starts by preparing students for a debate. This usually takes about a week and a half in which students come up with arguments and search for references on the internet to back up their arguments. After the debate, they are taught how to write a three-body paragraph argumentative essay. This is usually covered in about three weeks and no references are provided at this stage. Simple topics are usually given to students like whether students should work part-time or not, whether students should study abroad or in their own country, whether life in a city is better than life in a village, etc. Students are then taught how to summarise, paraphrase, and quote from texts. They are also taught how to cite their sources. Many short newspaper articles from The Independent, The Daily Star, The Guardian, etc., are provided in their handbook for this purpose. The topics are simple topics that students can understand and identify with like the health hazards of energy drinks, the danger of third-hand smoking, advertising to children, children's addiction to McDonald's, etc. Students are usually asked to read an article, summarise it, then paraphrase a part of it, and quote a sentence using a proper reporting verb. This is usually covered in about three weeks.

Then students are introduced to the process of writing a documented essay based on references provided by the teacher. They are also taught how to write a reference list in addition to properly citing the references in the text. This takes about two to three weeks. In the last part of the course, each student has to choose a debatable topic, take a position, and write a short research paper in which they incorporate references to back up their positions. Students are usually encouraged to use newspaper articles and websites for organisations. They bring in their references to class and write the paper in class. In all the three major writing assignments, the argumentative essay, the documented essay and the short research paper, students are usually required to write two drafts. Students do a second draft for each of the three graded assignments after

getting written and oral feedback from their teachers that covers language, mechanics, sentence structure, word choice, ideas, unity and coherence.

Although the English 102 syllabus requires students to use CT skills, whether this is taught or not and how it is taught is left to the discretion of the class teacher. Although one could argue that writing a documented essay or a research paper requires the use of CT, students in this module do not get enough training in thinking critically and evaluating claims and sources. Based on the observation of the researcher and many