Data Analysis Procedures—Qualitative

In my qualitative analysis, I used thematic analysis to uncover trends in thoughts and opinions that could provide greater insight into the quantitative outcomes. In order to check the legitimacy and increase the reliability of the thematic analysis, I used interrater reliability tests, which I conducted with a research assistant.

4.6.2.1 Thematic Analysis

Thematic analysis is a method for examining interviewees' words in the transcript to identify key patterns of emerging elements within the data and then analyzing those elements

2 _{I attempted to use multiple linear regression analysis for self‐reported memory, but the result did not include any} 

(repeated keywords and phrases) to form them into codes (Braun & Clarke, 2006). Codes, in qualitative analysis, are defined as “tags or labels for assigning units of meaning to the

descriptive or inferential3 _{information compiled during a study” (Miles, Huberman, & Saldaña,}

1994). The set of codes makes up a codebook that the investigator compiles to enable him/her to analyze the corpus of transcript data.

As noted above, the process of thematic analysis enabled me to extract complex relationships among students’ descriptions and ideas about their experiment experience. I conducted line-by-line coding, in which I looked at every line and labeled everything I believed to be significant. For the coding process, I used a software package, QSR NVivo 10, that allows the user to easily label quotations in the interview transcripts and attach them to a code or codes. The term quotation can refer to a word, a phrase, a sentence, or a group of sentences. To generate the codebook for the whole qualitative analysis, I followed a series of six steps. First, I used a professional transcribing service to produce the transcripts. Second, I found the sample from which to generate the preliminary codebook. Third, I generated the initial codebook from the sample that I had selected. Fourth, I reviewed and revised the code in the context of the data, following thematic analysis steps developed by Braun and Clarke (2006). Fifth, on the basis of the initial data, I conducted interrater reliability tests on the preliminary codes with another researcher to validate the legitimacy of the initial codebook. Lastly, I coded the remainder of the transcripts using the validated codebook. The thematic analysis enabled me to focus on what was significant and on common patterns that were occurring.

3 _{By inferential, I mean, for example, I am attaching the tag “difficult intervention” to expressions that I can} 

reasonably interpret as related to difficulty, e.g. (“intervention was kind of hard to follow” or “that intervention  was easier to get distracted.”) 

4.6.2.1.1 Audio Transcribing

Audio interview data were transcribed by a professional transcribing service called Rev. I strictly confined the basis of the code to the words as transcribed by the professional transcribers. The transcripts that were received did not contain suprasegmental information (pauses, emphasis on certain words, intonation, etc.).

4.6.2.1.2 Choosing the Sample

In choosing the sample to make the preliminary codebook, I wanted to extract the codes from samples of the students’ transcripts that reflected all the possible online experiences. To do so, I looked at how to categorize the transcripts so that all the experiences were represented. To get a representative sample, I divided the transcripts into 8 subgroups by what path of

interventions they received and whether or not they had the writing reflection intervention, as shown in Table 4-3 below. I wanted to make sure that I got a transcript from each of the subgroups that I would get by dividing the whole group up according to these categories.

Table 4-3: Eight Subgroups by Types of Interventions Received Writing Reflection No Writing Reflection A-V-T A-V-T V-T-VT V-T-VT T-VT-A T-VT-A VT-A-V VT-A-V

I then took two random samples from each subgroup because I estimated that only one would probably not adequately represent each subgroup. Therefore, I generated the preliminary codebook from transcripts provided by 16 students.

4.6.2.1.3 Generating a Preliminary Codebook from Samples

As preparation for analyzing the whole corpus, I generated a preliminary codebook from the transcripts of the initial selection of 16 transcripts. On the basis of Braun and Clarke (2006), I first created a 6-step procedure for analyzing the transcripts. I then used thematic analysis to analyze the initial transcripts, as shown in Table 4-4 below.

Table 4-4: Six-Step Procedure for Analyzing the Transcripts Thematic Analysis Steps

(Braun & Clarke, 2006) How I Performed the Data Analysis Step Result/Output 1. Familiarizing myself

with the data

Read and re-read a second time the sample transcribed data. Made notes about words or phrases occurring in a particular pattern

Log of preliminary ideas

2. Generating initial codes Coded recurrent or unusual features of the data in a systematic fashion across the entire sample data set

Log of data relevant to each preliminary code

3. Searching for themes Organized codes into potential themes

Log of data relevant to each potential theme

4. Reviewing themes Checked to see whether the themes worked in relation to the coded extracts

Checklist of reviewed themes

5. Defining and naming

themes Iterated the analysis to refine the specifics of each theme List of clear definitions and names for each theme

6. Producing the report4 _{Selected examples relating the}

analysis to the research question and literature (the final analysis of selected extracts)

Report on the analysis and the codebook

In the following paragraphs, I provide a description of the steps of my application of thematic analysis to my corpus:

4 _{The 6th step is reserved for the analysis including the remainder of the transcripts.} 

Step 1: Familiarizing myself with the data. I read the selected 16 transcripts (as noted above)