The aim of this pilot study was to ascertain the suitability of the measures selected for evaluating ThinkSmart. At this stage it was to improve the test measures. The pilot was able to provide an insight into the suitability of the measures selected to evaluate ThinkSmart. This was important as to date such an approach of using standardised measures had not be used.
7.3.1METHOD
7.3.1.1 Design
The young people completed three measures; Myself-As-A-Learner (Burden, 2000), Rosenberg's Self-Esteem Scale (Rosenberg, 1965) and The School Engagement Scale (Fredrick, Blumenfield, Friedel and Paris, 2003).
7.3.1.2 Sample
In total thirty nine young students (twenty two boys and seventeen girls) from four secondary schools in Herefordshire and Worcestershire returned their informed consent form and completed all three measures. All participants were from Year 9 so aged between 13 and 14 (mean age 166.74 months (3.6 SD)). The number of young people for each school site is detailed in table 7:1.
Table 7.1Sample Numbers for the Pilot of Outcome Measures at Each School Site
School A School B School C School D Total
Sample Size 5 6 24 4 39
7.3.1.3 Measures
The selection of instruments was based on the intended outcomes of ThinkSmart: improvement of self-perception, self-esteem and school engagement.
Myself-as-a-Learner Scale (Burden, 1998) is a self-report measure that assesses young people's perception of themselves as learners and problem solvers (see appendix 5). A student's ability to succeed in school is associated with the attributions they make about their success and failures. Other measures of self-
93 concept are evident in the literature such as those devised by Herbert Marsh and Susan Harter, these are however measures of general self-concept so comprise of a number of items increasing the time of administration (Burden, 1998). So not to interfere with the implementation of ThinkSmart, the measures needed to be quick to administer. Myself-as-a-Learner (MALS) was quick to administer and measured the aims of ThinkSmart. MALS was considered suitable as it is for use with the same target audience asThinkSmart; as the intended use of the measure was for students with low self-perception and has been reported as a useful tool to evaluate similar programmes to ThinkSmart.
The measure is deemed straightforward to administer with all the items being easy to understand for the intended participants. It is a twenty itemed self-report measure which the young people scored their answers from an A (very true) to an E (definitely not true); these were then converted into scores via the scoring sheet. The maximum score is 100 and the minimum score is 20. The measure is robust with both validity (face and construct) and internal reliability (alpha value .8) been established (Burden, 1998). Overall this appeared to be an efficient scale for measuring the multifaceted nature of self-concept (Burden, 1998).
Rosenberg’s Self-Esteem scale (Rosenberg, 1965) is a well-established measure of self-esteem (appendix 6). Self-esteem is thought to play a role in explaining educational progression as one of a number of psychological constructs, thus the measure was included in the ThinkSmart manual for the young people to complete at pre- and post--test. A wealth of research has attested this to be a reliable and suitable measure for the inclusion in the evaluation of ThinkSmart, with alpha values ranging between 0.78 and 0.90 (Rosenberg, 1965). It is a self-report measure that is comprised of ten items (five positively worded and five negatively worded). The scale can be administered as either a four, five or seven point Likert scale, for this study a four point scale was used; as this was what was included in the ThinkSmart manual to eliminate the neutral ground thus respondents had to provide an answer in either direction, positive or negative.
School Engagement Scale (Fredricks, Blumenfeld, Friedel and Paris, 2003) is an American measure of school engagement on three dimensions; behavioural, emotional and cognitive (Fredricks, Blumenfeld, Friedel and Paris, 2003) (appendix
94 7). ThinkSmart aimed to increase the young people's engagement with school because research proposed disengagement with school was associated with low achievement. This scale suggested there are three components to understanding school engagement. The behavioural engagement measured by five items (alpha .75) related to the idea of participation at school, social activities and it was included in the scale as this type of engagement is deemed necessary for positive academic outcomes. Emotional engagement measured by six items (alpha .83) focused on relationships with teachers, peers and school and how these emotional relationships influenced engagement. Cognitive engagement measured by eight items (alpha of .82) discussed the investment with school work. The items for the measure were constructed from a number of measures as well as adding in new items (see Fredricks, Blumenfeld, Friedel and Paris, 2003). The scale comprised of nineteen statements on a 5-point Likert scale from strongly disagree to strongly agree.
The measure was developed for a project that explored children's (aged 8- 11) engagement in school. When piloted the measure demonstrated good face validity and adequate internal consistency measured by Cronbach's alpha .75 to .83. This project piloted the measure with an English sample of pupils two years older than the original sample used for the construction of the measure to deem whether the measure was suitable with this age group. It was a newly devised scale, so unlike the other two measures there was little empirical research.
7.3.1.4 Procedure
Permission was sought for the young people to participate in the study.Letters explaining the study, their child’s right to withdraw and assurance of anonymity and confidentiality were sent to parents/carers. Those young people who returned their parental consent forms were informed about the research and asked if they wished to participate. Those who agreed completed the three measures at the same time in a classroom setting. At the end participants were debriefed about the project and asked to provide feedback on the questionnaires.
95 7.3.2RESULTS
7.3.2.1 Descriptive Statistics
Table 7:2The Means and Standard Deviations for Pilot of Study Measures
Measure Means and Standard Deviations Myself-As-A-Learner Scale (MALS) 65.05 (9.03)
Self-Esteem Scale 16.64 (4.1)
School Engagement Scale 56.59 (8.5)
The initial standardisation of the MALS scale provided a benchmark of scores between 60 and 80 to represent the average scores expected for students between the ages of 11 to 16. Table 7:2 shows the mean score to be close to the lower end of the benchmark with a small variation in scores understood by the standard deviation. Thus proposing the measure might not be suitable. This is also similar for the School Engagement Scale, in which the mean score is relatively low. These scores however might be due to the sample selected having low levels of self-perception and school engagement thus the reliability of the measures was also assessed.The mean for the Rosenberg self-esteem scale demonstrate the mean score to be above the score of 15 which is considered to be the cut off for low levels of self-esteem, and the scores are concentrated due to the small standard deviation.
7.3.2.2 Reliability of Measures
To assess the appropriateness of the outcome measures for the main evaluation, the internal consistency of each scale was examined using Cronbach's alpha. Internal
consistency measures whether the items ‘hang well together’, so they are all
measuring a similar construct. MALS and Rosenberg's self-esteem scale reported good internal reliability- see table 7.3. The School Engagement Scale reported an alpha level of .74 just above the standard benchmark of .7 used to determine the reliability of a measure (Streiner and Norman, 2008).
96 Table 7:3 The Cronbach's Alpha for Pilot Measures
Measure Cronbach’s Alpha Myself-as-a-learner (MALS) .82
Self-Esteem Scale .82
School Engagement Scale .74
The evaluation of ThinkSmartwas not to interfere with the implementation of the intervention, as it was an independent evaluation to improve ecological validity. However, feedback from the young people indicated that the completion of three measures was burdensome, and a number of non-responses were evident. Missing data can make it difficult to make valid statistical inferences as non-responses are systematic (Jones, 1996). The root cause must therefore be investigated. Two measures, MALS and School Engagement Scale, were highlighted by the young people. The School Engagement Scale was a newly devised scale that Fredricks et al., (2003) noted the scale may need adapting for older children. The young people struggled with the completion of the measure, as a result this measure was not included in future studies. Furthermore, the measure MALS was standardised with students in year 7 and 8, Burden (2010) reported that with older students the scale may be less reliable. These results suggest for this project to be the case, as the scores are close to benchmark. Thus a more suitable measure of motivation and engagement to include in the evaluation of ThinkSmart was needed.