Data Collection: Quantitative

Much review and research went into the selection of assessments for the

quantitative data collection. Numerous factors influenced these decisions such as availability of the assessments, financial costs to the researcher, time consumption in delivery and analysis of findings, relevance to the aims and research questions, language and suitability in design, validity and reliability, as well as commonality of use in research and/or interventions of this type.

The quantitative data was collected on a cohort-by-cohort basis and in three specific areas: reading and literacy, social and emotional, and behavioural. No priority was given to quantitative data versus qualitative as previously discussed in the section 4.3.1 and both were collected concurrently; however, the times and testing conditions of the assessments were different for cohorts 1, 2, and 3.

Cohort 1: initial assessments were administered on the first day of the programme as a whole group at the facility provided during the first hour by the researcher and the third party assistant; post follow up assessments were administered within 30 days of the programme completion at the participant’s school on a one to one basis with the researcher and student in a private room.

Cohort 2: initial assessments were administered on the first day of the programme as a whole group at the facility provided during the first hour by the third party

assistant only (the researcher was not present); post follow up assessments were administered within 30 days of the programme completion at the participant’s school on a one to one basis with the researcher and student in a private room.

Cohort 3: initial assessments were administered on an individual basis with the student and researcher in a private room at the participant’s school two weeks prior to the beginning of the programme; post follow up assessments were administered immediately following the final lesson due to termination of the school year/access. This was done as a whole group at the facility provided during the final hour of the programme.

4.5.2 (A) Reading and Literacy Assessments Fluency (pre and post)

Research (Chard, Vaughn, & Tyler, 2002; Kuhn & Stahl, 2000; Samuels, 2002; Dowhower, 1991) has determined that reading fluency is a crucial element of learning to read and often acts as a link between the two major components of reading: word decoding and comprehension. It connects accuracy and automaticity in decoding to comprehension through prosody. It is suggested that three distinct areas of fluency be considered when assessing: decoding accuracy, meaning the ability of readers to decode words accurately in a selected text; automaticity, meaning the ability of readers to decode words in a text with minimal use of attentional resources; and lastly, prosody, meaning the ability of readers to appropriately use phrasing and expression (Rasinski, 1985, 2003).

Informal reading inventories or IRI’s are commonly used to assess fluency by focusing on accuracy determined by the percentage of words a reader can read correctly. However, for the sake of this research, IRI’s alone were deemed too long (some taking up to two hours to deliver) and too exhaustive (requiring the reader to attempt numerous word lists and passages orally while being tested for comprehension in each passage). With this is mind, it was decided to use a Curriculum Based Measurement (CBM) or Oral Reading Fluency (ORF)

which time the administrator of the assessment marks for uncorrected errors and then counts the total number of words read correctly per minute or WCPM (Deno, 1985). The procedures for administration of the CBM/ORF tests are as follows:

1. A passage of approximately 250 words written at the student’s grade level (Year 9) was selected and a Fry Readability test conducted to determine grade appropriateness. Please see Appendix 6 for the text used.

2. The students were separated on an individual basis and asked to read the text in a normal rate and volume for one-minute/60 seconds.

3. During this time, the researcher marked any uncorrected errors including omissions, mispronunciations, substitutions, reversals, or words needing assistance by the researcher after two to three seconds pause.

4. Accuracy was then determined by dividing the number of words read

correctly per minute by the total number of words read (WCPM + any errors) to get a percentage.

Reading (pre and post)

No standardized reading assessment was administered prior to or following the intervention; rather, data regarding reading was collected from the participating schools on each of the students in a method to which they chose. Only one of the two participating schools in Cohort 1 provided this information and did so in the form of reading ages. Cohort 2 schools provided reading marks from the beginning of school to final marks. Cohort 3 schools provided reading ages from the

beginning of school to final marks.

Chapman Reading Self-Concept Scale (pre and post)

The Reading Self-Concept Scale (RSCS) by Chapman & Tunmer (1995) was chosen as the assessment for determining pre and post reading self-efficacy scores of the participants. This was done so due largely to its condensed version aimed specifically at measuring the three main important aspects of the reading sub-component of academic self-concept: perceptions of competence in reading, perceptions of difficulty with reading, and attitudes towards reading. Although this

assessment is designed to target children from Year 1 to Year 5, the ease of administration and the language used in its design made it a more suitable choice for this research project.

In addition, the statistical analysis provided by the RSCS reveals a “strong positive correlation between the competence and attitude factors…lower positive

correlations were found between the difficulty and competence factors and the difficulty and attitude factors” (Chapman & Tunmer, 1995, p. 1) which begins to develop around seven years of age. The indications are that “reading self-concept is more likely to be a consequence than a cause of reading performance” (p. 2). This is particularly important to the qualitative analysis of impact of the intervention on reading, reading self-efficacy, and overall academic self-concept discussed in Chapter 6.

Similar to the Fluency assessments, every attempt was made to replicate the same testing conditions, environments, and times for each cohort. Cohorts 1 and 2 were given the RSCS on the first day of the intervention at the primary location of

programme delivery. The students were not isolated, rather sat at tables of two. The test questions were read aloud to the whole group, and students were asked to answer honestly. A second administrator was available during the test to aid students should they have had questions. This administrator also ensured the students answered as best they could, not leaving any blanks, or writing the same number/response down for each question. This process was repeated for Cohort 3, but students were given the test individually in a private room prior to the

beginning of the intervention at the participant’s school. For follow up assessments post intervention, Cohorts 1 and 2 were given the tests individually at their schools within 30 days of completion. Cohort 3, however, was given the post-test

immediately following the completion of the intervention due to the termination of the school year/access. This was done as a whole group as described above.

4.5.2 (B) Social/Emotional Assessments

Pearson Resiliency Scales for Children & Adolescents

The Pearson Resiliency Scales for Children & Adolescents was chosen to assess social and emotional changes before and after the intervention. There are a variety of operational definitions for resiliency varying from competence, self-esteem, optimism, and more. However, the PRSCA defines resilience as “the ability to weather adversity or to bounce back from a negative experience” (Prince-Embury, 2007, p. 1). In addition, the Pearson Scales were designed “to systematically identify and quantify core personal qualities of resiliency in youth, as expressed in their own words about their own experience” (p.1). This was quite important in selecting an assessment that would effectively examine the variety of social and emotional experiences of the participants as well as the numerous factors that contribute to these with the focus on the participants’ own words and experiences. Specifically, the purpose of the Pearson Scales “is to provide theoretically and empirically sound assessment of core characteristics of personal resiliency in children and adolescents (ages 9-18) that are easily communicated to them and their care givers for the purpose of education, screening, prevention, and

counselling” (p.1). The Pearson Scales acknowledge the value of environmental and external forces that youth bring as highly influential to their overall well being; what individual characteristics help them to cope and adapt to these adversities is the focus of the assessment.

There are three core areas identified as measurable constructs of resiliency: Sense of Mastery, Sense of Relatedness, and Emotional Reactivity.

Sense of Mastery (Strengths)

A sense of mastery in adolescents is one that provides them the opportunity to “interact and enjoy cause and affect relationships in the environment” (p.9). Three personal attributes have been identified as the main contributors to sense of mastery:

Optimism, meaning “a positive attitude about the world/life in general and about

the individual’s life specifically, currently, and in the future” and as “attribution style, positive self-esteem, and perception of control” (p. 9).

Self-efficacy defined as the sense that one can master his or her own

environments (including perceived self-efficacy), individual approaches to obstacles or challenges, and the motivations and actions spent towards these adverse situations.

Adaptability or flexibility is the ability to consider different opinions in problem

solving, the capacity to think alternatively, learning from his or her mistakes, and asking for help when needed.

The Sense of Mastery Scale is a 20 item self-report questionnaire written on a Year 3 reading level with response options ordered on a five point Likert Scale: (0) Never, 1 (Rarely), 2 (Sometimes), 3 (Often) and 4 (Almost Always). These were conducted pre and post intervention.

Sense of Relatedness (Relationships)

The Sense of Relatedness Scale is “based on the assumption that the capacity to be in a relationship is a basic human function as a social organism…as feeling securely connected to individuals in a social context” (p. 11). During the

development of this scale, four areas were determined to be components of relatedness:

Trust, defined as the degree to which others are seen as reliable and accepting as

well as the degree to which and individual can be genuine in these relationships.

Support, meaning one’s personal belief that he or she has others to whom they

Comfort, described as the extent to which a person can be in the presence of

others without discomfort or anxiety.

Tolerance, as a person’s belief that he or she can safely express differences

within a relationship.

The Sense of Relatedness Scale is a 24 item self-report questionnaire written on a Year 3 reading level with response options frequency based on a five point Likert Scale: (0) Never, 1 (Rarely), 2 (Sometimes), 3 (Often) and 4 (Almost Always). These were conducted pre and post intervention.

Emotional Reactivity (Managing Vulnerability)

Emotional Reactivity is viewed here as “pre-existing vulnerability, arousal, or threshold of tolerance to stimulation prior to the occurrence of adverse events or circumstances. Relative reactivity may have physiological bases, such as temperament, genetic predisposition, learning disability, physical impairment, or congenital anomaly” (p.12). The scale’s design was based on the assumption that children and adolescents might be better able to report their experiences of

emotional reactivity versus their emotional regulation abilities. It conceptually represents sensitivity, one construct of emotionality, and two constructs that represent the outcome of the regulatory processes of recovery and impairment. This scale therefore does not presume to assess emotional regulation directly, but rather the degree to which youth experiences maintaining an even keel when emotionally aroused (p.12). As said, the three areas of assessment are:

Sensitivity is the threshold for reaction and the intensity of the reaction. Six items

ask to assess “how easy it is for he/she to get upset” and how upset he/she gets. The word upset was chosen intentionally to indicate a state of arousal or

disequilibrium without specifying one emotion.

Recovery is the ability to cope and move on from emotional arousal or disturbance

recover when upset or angry. The youth indicates the relative frequency with which that recovery time is true for him/her.

Impairment is the extent to which the adolescent is able to maintain an emotional

balance when challenged. Ten items ask to report the frequency with which some impairment occurs related to being upset. Examples are losing control, making mistakes, not thinking clearly, and getting into trouble.

Overall, the Emotional Reactivity scale is a 20-item self-report questionnaire written at a Year 3 reading level with response options ordered on a five point Likert

Scales as are the other sections. However, lower scores on the ER scale are indicative of resiliency (desirable) and high scores are indicative of vulnerability (undesirable). These were conducted pre and post intervention.

4.5.2 (C) Behavioural

Behavioural data was collected in four areas: school behavioural points/records, school achievement points/records, intervention Weekly Progress Reports, and school attendance records. However, during analysis, it was decided to use the Weekly Progress Reports and school attendance records in the qualitative section versus quantitative.

Behavioural Points/Records

No standardized behavioural assessment was administered prior to or following the intervention; rather, data regarding behaviour was collected from the participating schools on each of the students before the intervention, during the intervention, and following the intervention. The exception to this was Cohort 3, as the school year ended at the same time the intervention ended, therefore no post behaviour records were provided.

Each school used the same points based system for recording behaviour issues; students were assigned points based on the severity of the action of misbehaviour.

L1’s were issued by the class teachers and tutors for incidences including low level

disruption in the classroom, late to lessons, off task, no equipment, failure to complete homework, etc. This included any detention. Generally, 1-2 points.

L2’s were issued by class teachers, tutors, and Head of Year for incidences such

as repeated disruption to learning, repeated refusal to follow instructions, use of inappropriate language, failing to attend a break or lunch time detention, etc. This included any in school exclusion. Generally, 3-5 points.

L3’s were issued by tutors, Heads of Year, and SLT for any serious or on-going

breaches of the school behaviour policy, searing at a member of staff, violence, bullying, etc. This included any out of school exclusion or suspension. Generally, 10 points.

Achievement Points/Records

No standardized assessment on achievement was administered prior to or

following the intervention; rather, data regarding achievement was collected from the participating schools on each of the students before the intervention, during the intervention, and following the intervention. The exception to this was Cohort 3, as the school year ended at the same time the intervention ended, therefore no post achievement records were provided.

Each school used the same points based system for recording achievement; students were assigned points based on the action of merit; for example, a student could receive one point for extra efforts in class or settling to tasks quickly,

contributing to class discussion or making a positive influence. A student might receive five merits or points for participating in an after school activity, volunteering for extra work, or showing extreme compassion towards another student. The merits or points were given by subject teachers, tutors, Heads of year, or any other faculty member and were either one point or five points respectively.