Mixed Methods Research Design and Approach
Mixed methods research methodology was used to determine if there was a measurable change in student assessment data of those students who used the myON® reader program at ABC Charter High School for school years 2012-2013 and 2013-2014. For the purposes of this study, pretest and posttest quantitative assessment data were evaluated to determine if there was any change in reading proficiency rates from 2013 to 2014. The qualitative portion of this research was conducted through an in-person interview with each of the teachers who helped implement the myON® reader program. These data were gathered to gauge their opinions of the program; as well as any changes in student attitude, motivation, or other factors towards reading in general and in other coursework.
Mixed methods design is typically used when both quantitative and qualitative data are available and when used together they would make for a stronger study outcome (Creswell, 2012). Quantitative research evaluates numerical data to summarize results, whereas qualitative researchers work with data that are more likely to be narrative or verbal in nature (Lodico et al., 2006). The combination of these two types of data gave me the opportunity to gain better insight as to whether the myON® program has been successful not only for student assessment, but for developing students’ attitudes as well. This method allowed me to overcome the intrinsic weaknesses using quantitative and qualitative methods separately might have had (Creswell, 2012).
Justification for Using Mixed Methods Design and Approach
There was not an opportunity to manipulate the independent variable for this study, making the mixed methods with a non-experimental research design the best approach (Creswell, 2012). The quantitative assessment data were collected once a year, using the Read Fluency Benchmark Assessor, and was provided to myself for review. Also provided were data available from the myON® reading program on each student’s
reading activities throughout the year. The qualitative data were gathered through
personal interviews with the teachers. The combination of the quantitative and qualitative data gave an in-depth understanding of the results and the benefits to the students.
Strategy for Data Collection
For the purposes of this study, I collected the data in an explanatory sequential mixed methods design. The explanatory design is used when collecting quantitative data first and placing a priority on that data set when it comes to evaluation (Creswell, 2012; Lodico et al., 2006). The qualitative data were collected and analyzed after the
quantitative data had been received and examined. Reviewing the quantitative data first allowed me to use the qualitative portion of the study to probe more deeply into results that were found in the quantitative part of the study (Creswell, 2012).
Forms of Data Collection
The quantitative data were obtained through regularly scheduled assessments, which are given at the beginning of each school calendar year. Other quantitative data were collected from the myON® reader program itself as it tracked students interactions
with the program. The qualitative data were obtained through a semi-structured interview that I conducted with the staff of ABC Charter High School.
Summative Evaluation
This study consisted of a summative evaluation of the myON® reader program that will be provided to the principal of the school at study completion. This information will allow the principal and school staff to determine if the program was having the desired effect for the students. Summative data are collected to measure an outcome of the implementation of a program, in this case myON® reader (Lodico et al., 2006). This type of data was used because its purpose was to determine if goals or benchmarks have been met by the end of the program year(s) (Lodico et al., 2006).
Setting and Sample
The charter school where I conducted the research was small, with 179 students in Grades 9 through 12. Of the 179 students, 39 had been using the myON® reader program regularly over the previous 2 years. This study will not be representative of the general population, so convenience sampling was used. All of the students’ myON® reader viable program data were analyzed.
Data Collection
The first phase of the exploratory design was to gather the quantitative data and then follow up with the qualitative data (Lodico et al., 2006). The quantitative data were analyzed first in order to help guide the writing of the interview questions. This analysis made sure that any questions that arose in the review of the quantitative data could be elaborated on as part of the qualitative portion. The following will go into more detail of
what instruments I used to collect these data, what was measured, and how that measurement took place.
The quantitative data came from two sources: regularly scheduled Read Fluency Benchmark Assessor and the myON® reader program itself. The data received were only for students that used the myON® reader program throughout the year. Students that did not use the myON® program were removed from both the assessment and the myON®
data being provided to me. The program provides data on individual students, entire classrooms, or schools. These assessments are conducted at the beginning of the school year. The myON® reader program tracks what the students read, at what level they are reading, frequency, and progress throughout the year.
The data from the Reading Fluency Benchmark Assessor and myON® program
were made available to me via recordable removable media. The removable media was password protected, as was my computer, in order to prevent others from gaining access to the data. The identity of the subjects were not provided to me in order to maintain confidentiality. A key coding process was used to hide the identity of the subjects, which was done by the principal at the school prior to providing the data to me. I never had access to the key at any time in the process.
For the qualitative portion of the study, a semi-structured interview was
conducted with the staff to get feedback on their perceptions of the program. Because the school was so small, all five of the teachers that interacted with the students during this period were asked to volunteer for an interview. Of the five asked, two agreed to participate in the interview but requested to be interviewed at the same time. The
motivation for the interview was to find out if they saw any changes in student attitude, motivation, and confidence towards reading as they used this program. A semi-structured interview has a set of questions to lead the interview but that do not dictate the
conversation as it pertains to the study (Lodico et al., 2006). The interview protocol used can be found in the appendices (See Appendix C). In order to create the interview
questions, I first made a list of questions I thought would get the information that I desired from the teachers. However, before the interviews were conducted, the
quantitative data were reviewed to determine if there were any trends or themes that may have needed clarification or further insight when speaking with the staff. Based on this determination, the questions were adjusted where needed, which were minor. The
interview protocol provided in Appendix C includes these changes and is what I used for the interview. The interviews were audio recorded to allow me to participate actively in the conversation more freely without the fear of missing important comments and information.
Data Analysis
Quantitative Data Analysis
Microsoft Excel and SPSS statistical software were used to help with analysis of the data from both the assessor and the myON® reader program. Descriptive statistics were used to examine the data, which include central tendencies, variability, and relative standings (Table 4). These were used in order to determine the direction of the changes in the data provided by the assessor and the program for each student (Creswell, 2012). The goal was to gain insight into the myON® reader program and the perceived relationships
between students’ use of the program and differences between their pretest and posttest data.
In order to interpret Table 4, please refer to the following bulleted list:
Fall_2013 – The results of the Reading Fluency Benchmark Assessor for Fall 2013.
Fall_2014 – The results of the Reading Fluency Benchmark Assessor for Fall 2014.
Hours_Read – The number of hours the student spent reading over the one year period
Diff_Accucess_Score – The difference between the Fall_2013 (pretest) and Fall_2014 (posttest) data
Lexile_2013 – The Lexile or reading level the student was at when they first started using the program in 2013
Lexile_2014 – The Lexile or reading level the student was at when they started the school year in 2014
Lexile_Diff – The difference between Lexile_2013 and Lexile_2014
Table 4
Descriptive Statistics
N Minimum Maximum Mean Std.
Deviation Fall_2013 39 3.72 12.00 9.5477 1.80261 Fall_2014 39 5.84 12.50 10.9391 1.42984 Diff_Accucess_Score 39 -3.91 5.30 .8454 2.07805 Hours_Read 39 12.00 64.00 34.8718 14.01912 Lexile_2013 39 410.00 1185.00 926.7949 167.89912 Lexile_2014 39 600.00 1300.00 974.2308 143.33223 Lexile_Diff 39 -45.00 200.00 47.4659 53.09961
Table 4 provides the descriptive statistics used for the study. There were 39 students that used the myON® reader program regularly, and these student’s pretest and posttest assessment scores were evaluated. The students mean assessment scores between Fall_2013 and Fall_2014 increased by 1.39 points indicating there may have been an improvement over the year. This improvement is also indicated by the mean of the Diff_Accusess_Score, which is the difference between the pretest and posttest, had an increase of 0.8454. The mean reading level showed an increase between Lexile_2013 and Lexile_2014 as well with a positive change of 47.44 in students’ Lexile score (reading level). The information provided in Table 4 does give an indication that there was an
overall positive trend in that the means were all positive. As for the standard deviation these numbers do not necessarily indicate a positive or negative outcome they do,
however, give us an indication of how spread out the data is around the mean or average. In the case of this research study, it does not do much more than that.
The reading assessment tool being used, Reading Fluency Benchmark Assessor, has gone through many years of validity testing and reviewed for test-retest reliability (Reading Fluency Benchmark Assessor - Technical Information, 2009). The original passages used in this assessment tool were evaluated in Washington State between 1998- 2000 in order to determine concurrent and predictive validity (Reading Fluency
Benchmark Assessor - Technical Information, 2009). These eventually became what are referred to as the anchor passages for all future versions. Field tests were also conducted in order to confirm that the passages were properly written for the grade level intended, which led to another set of reliability and validity test being conducted in the 2002-2003 school year in seven states (Reading Fluency Benchmark Assessor - Technical
Information, 2009). The Read Fluency Benchmark Assessor is a tool that is used to assess a student’s fluency by providing three passages for each grade level 1-8 (Reading
Fluency Benchmark Assessor, 2009). The program uses three fiction and nonfiction passages for each grade level. The student reads these passages aloud and the teacher records errors during reading. These passages have been leveled using several readability formulas as well as extensive field testing. The field testing was conducted to check reliability and validity of each passage to ensure that they are providing results at the proper grade level (Reading Fluency Benchmark Assessor, 2009). This instrument is both
a norm-referenced and self-referenced. I will assume that the data from this source are accurate and provide a strong view into any changes in student reading proficiency.
The Reading Fluency Benchmark Assessor data and the myON® reader program data were provided to me by the principal at ABC Charter High School. Data use
agreement can be found in Appendix D. In order to maintain anonymity, the names were removed and a unique code was assigned to each student, which carried throughout all of the quantitative data received. Once the relevant data were provided to me via Microsoft Excel, I transferred it into the SPSS program for evaluation.
Pretest and Posttest Analysis
In order to evaluate the significance of the changes between the pretest and posttest data, the paired-samples t test was used to address the research question 1. The paired-samples t test is often used to compare two groups that can be related. In this case, the participants are members of both groups of data being studied (Lodico et al., 2006). This case can also be referred to as a repeated measures design: pretest and posttest.
Research Question 1. How significant was the difference in pretest and posttest reading achievement data for those students using the myON® reader program? In order
to answer this question, I compared the mean student scores from the Reading Fluency Benchmark Assessor pretest and posttest data to determine if there was a significant difference between the two. In order to determine this, a paired-samples ttest was
conducted to compare the pretest and posttest data. Table 5 shows the results of the t-test. The data showed, with a 95% confident level, that there was a significant difference between the pretest (M = 9.55, SD = 1.80) and posttest (M = 10.94, SD = 1.43)
conditions; t(38) = -2.541, p = 0.001) (Table 5). In other words, because the Sig. (2- Tailed) value is below .05 it can be concluded that a statistically significant difference exists between the pretest and posttest data. Meaning the students using the myON® reader program did increase their reading assessment scores at a significant level. Table 5
Summary of Paired-Samples t Test Comparing Pretest and Posttest Data
Paired Differences
95% Confidence Interval of the Difference
Mean
Std. Deviation
Std. Error
Mean Lower Upper t df
Sig. (2- tailed)
Pair 1
Pretest-Posttest -.84538 2.07805 .33275 -1.51901 -.17176 -2.541 38 .015
The quantitative data analysis of Questions 2 and 3 was conducted using the statistical test Pearson’s correlation coefficient. The Pearson’s correlation coefficient was used because there is only one independent variable being studied (Creswell, 2012). An independent variable is a change in an educational practice or approach with the
expectation that it will affect an outcome (Creswell, 2012; Lodico et al., 2006). The myON® reader program is the independent variable in this study, as it is the educational program that was implemented with the goal of improving student reading proficiency. The correlational research strategy is used to find patterns or relationships between two or more sets of data (Creswell, 2012). This strategy allowed me to determine if there was a correlation between the pretest and posttest assessment as well as data received from the myON® reader program itself.
Research Question 2. What level of correlation, if any, was found between the reading proficiency levels and quantity of reading students were engaged in at ABC Charter High School that used the myON® reader program? To answer this questions, a Pearson’s correlation coefficient was used to determine if there was a relationship
between the hours the students spent using the myON® reader program and the difference between their pretest and posttest assessment data. This evaluation was accomplished using SPSS software.
Figure 5 scatterplot shows a general upward trend, which indicates that there is some level of positive correlation between the difference variables. Once this was determined, I began the process of investigating the numbers further. Table 6 shows the outcome of that investigation.
Re
adi
ng H
ours
Reading Assessment Scores
Figure 5. Scatterplot of reading hours compared to assessment scores.
Table 6
Correlation Between Hours Read and Difference in Pretest and Posttest
Hours_Read Diff_Pretest_Posttest Hours_Read Pearson Correlation Sig. (2-tailed) N 1 39 .527** .001 39 Diff_Pretest_Posttest Pearson Correlation Sig. (2-tailed) N .527** .001 39 1 39
** Correlation is significant at the 0.01 level (2-tailed).
The information in Table 6 shows that there is a positive correlation between the two variables, r = 0.527, n = 39, p = .001. In this case, the r value of .527 indicates that
0 10 20 30 40 50 60 70 -6 -4 -2 0 2 4 6
there is a moderate correlation between the variables (Lodico et al., 2006). The p value of .001 indicating the relationship is statistically significant which gives a 1 out of 100 opportunity of getting a significant correlation due to chance (Lodico et al., 2006).
Research Question 3. What level of correlation, if any, was found between the reading material level of difficulty and students’ reading proficiency levels of those students that used the myON® reader program? A Pearson’s correlation coefficient was
used for question 3 to determine if there was a relationship between the 2014 Lexile scores (reading level) and the 2014 reading assessment data. SPSS software was used to compute this as well. Just as it was done for question 2, a scatterplot of the data was created to determine if there appeared to be a correlation between the Lexile level and a student’s reading assessment scores. Figure 6 does show a positive slope indicating a correlation, which lead me to proceed with analyzing the data that was provided in Table 7.
2014 L exi le S core
2014 Reading Assessment Score
Figure 6. Scatterplot of reading assessment scores compared to Lexile scores.
Table 7
Correlation Between 2014 Reading Assessment Scores and Lexile Level
Lexile_2014 Fall_2014 Fall_2014 Pearson Correlation Sig. (2-tailed) N 1 39 .676** .000 39 Lexile_2014 Pearson Correlation Sig. (2-tailed) N .676** .000 39 1 39
** Correlation is significant at the 0.01 level (2-tailed).
In order to look more closely at the perceived correlation between the students Lexile rating and the students reading assessment scores a Pearson Correlation
500 600 700 800 900 1000 1100 1200 1300 1400 5 6 7 8 9 10 11 12 13
Coefficient was used to measure this relationship. Table 7 shows the outcome of these calculations. The numbers show that there was a positive correlation between the two variable, r = 0.676, n = 39, p = 0.000 which indicates a better than moderate correlation between the two variables. This means as one variable changes there was a measurable change in the same direction in the other variable (Lodico et al., 2006). The results suggest a positive relationship between the Lexile score and the difference between the pretest and posttest scores.
Qualitative Data Analysis
The qualitative data was obtained through one interview with the two teachers at ABC Charter High School. The interview was semi-structured in nature. A semi-
structured interview has a set of questions to guide the interview but does not dictate the interviews direction allowing me the freedom to probe deeper into their perceptions of the program by asking relevant questions based on active conversation (Lodico et al., 2006). I developed the questions for this interview early on in this study. However, once I reviewed the quantitative data, I looked at the list of questions to make sure that they were aligned with the results of the quantitative data analysis. I wanted to make sure that the line of questioning was going to be an additional piece of information to use in the triangulation process of reviewing the data. The questions I had originally written were aligned very well with what was found quantitative data with some slight adjustments. I removed any reference to increases in assessment data as I knew the teachers had seen the data, and it was clear there was an increase in students’ assessment scores. I turned the focus completely to their opinion and perceived usefulness of the myON® reader
program itself. The data collected from this interview was recorded, transcribed, and then analyzed to formulate the findings.
The first step after the interview was to jot down notes to capture the key points I took from the interview. The next step was to listen and transcribe the recording. Once this was finished, the data was coded to pull out common themes. First, there was a preliminary analysis in order to get a sense of the data. This analysis allowed me to determine the best way to organize the data and decide if more data is needed (Creswell, 2012). Second, the data was coded using a model of coding similarly to the one presented