To address the limitations of the previous research evaluating outreach activities (as discussed in chapter four) a number of approaches were included in this research project to enhance the evaluation of ThinkSmart.
6.3.1COMPARISON GROUP
A comparison group was included in the research design as it is not best practice for all participants to receive the intervention being evaluated. However, this is not an approach evident in the evaluation of outreach activities. Gorard et al., (2006) noted the ignorance to the need for comparison groups in the evaluation of outreach activities. The inclusion of a control group in outreach activities was rare. This is due to the use of a comparison group being considered unfeasible and perhaps even unethical (HEFCE, 2006). It is thought to be inappropriate because if a programme had targeted with precision then there would be no suitable comparator school, area
86 or individuals with similar backgrounds (Hatt, Baxter and Tate, 2005). However without this, it is not possible to identify causal relation between outreach activities and educational progression (Spall, 2005; Tate and Baxter, 2006). It is also argued that it is unethical to involve human participants in interventions where efficacy has not been thoroughly tested.
Moreover, without the inclusion of a comparator group it is not possible to determine whether any changes reported in the outcome measures are attributed to their experience of the intervention. It is only possible to infer causality through the comparison of two controlled situations in which one group receives the intervention and the other does not (Mill, 1865). A research design that does not include a comparison group is of almost no scientific value (Campbell and Stanley, 1963). It is therefore essential to have at least two conditions in an experimental evaluation study; one group who receives the intervention and one who does not. At present the conditions of an experimental design are considered difficult to meet in a widening participation context (Chilosi et al., 2009). This research project aimed to demonstrate it possible.
Randomly assigning participations to conditions is considered to be the gold standard in experimental designs (Borman, 2002). This is however not always feasible especially in naturalistic studies such as this research project. A naturalistic study evaluates an intervention that is natural occurring. Randomly assigning participants can improve the internal validity of the research as well as eliminate the possibility of individual differences and group differences (Larkin and Thyer, 1999). Yet, control on participants and restrictions on the intervention may result in low ecological validity. It was important the results of this research project could be generalised and applied to real-life situations. Ecological validity describes how the results of a research project can be applied to real-world situations (Field, 2009). Higher ecological validity describes that the project is representative of the real-life world application, thus considered a naturalistic research design. Results from a research project that has evaluated an intervention in an artificial setting such as assigning participants to conditions or restricting what would be the natural implementation of an intervention can sometimes be irrelevant to understanding the area being explored (Schumuckler, 2001). Thus the methods of evaluation for this research project are to ensure high ecological validity. Therefore restrictions were
87 not put in place on the allocation of participants; the implementation of the intervention which was overseen by Aimhigher Herefordshire and Worcestershire, so the evaluation of ThinkSmart was independent.
Additionally as part of the evaluation a six month follow-up was included. This is because previous evaluations merely reported the short-term impact of outreach activities. The inclusion of a delayed follow-up was to determine the intermediate impact of ThinkSmart on recipients relative to the control group.
6.3.2SAMPLE SIZE
For the evaluation of outreach activities sample size was rarely considered. The main concern was that the evaluations were kept small to ensure they were manageable for practitioners (Passey and Morris, 2010). Research reports often noted a small sample size for the evaluation of outreach activities, which with a qualitative approach meant the point of data saturation was possibly never reached. Moreover this would mean the views of all the young people who had participated in Aimhigher activities were not accounted for (Kerrigan and Carpenter, 2009). For the qualitative approaches used as part of the process evaluation in this thesis a representative sample was required. Thus all participants, including the young people, Ambassadors, Aimhigher staff and school staff, were asked to participate in the process evaluation of ThinkSmart. This ensured a representative proportion of the participants were interviewed to ensure the views provided were reflective of the general consensus of the research population.
If inferential statistics are to be used, as was the case in this research project to analyse evaluation data, a priori power analysis (an assessment of statistical power carried out prior to a study being undertaken), should be conducted. A priori power analysis was carried out to ascertain the number of participants that should be recruited for the data to be meaningfully analysed statistically. Statistical power is the probability of a test to detect a statistically significant effect, if one exists (Field, 2009). To ascertain the minimum sample size required for this study, G*Power a software for power calculation was used, the estimated sample size required utilising G*Power was minimum of 32 participants in total. Using such software to determine the sample size before data collection places confidence in the results and enables
88 generalisation of the findings. However, the recruitment was outside of the researcher’s control.
The recruitment of the schools, young people and the Ambassadors was the role of the ThinkSmart Coordinator employed by Aimhigher Herefordshire and Worcestershire. The evaluation was independent to overcome the inherent biases of practitioners evaluating their own activities. The naturalistic nature of this project increased the ecological validity, thus improving the application of the research findings outside of the research context. This is of importance to improve current practices in the design and evaluation of outreach activities. An independent evaluation also enabled reliable and valid conclusions to be drawn.
6.3.3STANDARDISED MEASURES
Questionnaires were commonly used to evaluate outreach activities, yet these were devised at practitioner level. This questions the reliability and validity of these measures and thus whether the results reported were credible. Passey and Morris (2010) reported that the use of the standardised measures to collect data was too difficult. However standardised measures were used in this research project to measure the actual impact of the intervention. To determine the most suitable measures to evaluate ThinkSmart, the next chapter describes the pilot studies that were conducted prior to the main evaluation.