Evaluation of the programme

The evaluation of the programme incorporated both process and outcome components.

Evaluation of process aspects utilised a number of data sources. Firstly, students completed a transition programme evaluation questionnaire which asked them to rate different elements of the programme. Secondly, the researcher made contemporaneous notes of the behavioural responses of

the student participants; and notes were kept of the operation of the activities in each session. Thirdly, feedback comments were provided by the researcher, the students and parents. Finally, all of the sessions were video recorded. The first two data sources will be utilised in this report.

Data from the transition programme evaluation questionnaire relate to research question 2. With regard to research question 3, the focus was the researcher’s subjective view of the operation of the sessions. Analyses of the researcher’s comments in the ‘content and evaluation of sessions’ and the ‘observation of participants’ forms were undertaken (see section 2.2.3.2). A separate report, which will take the form of a multiple case study, will utilise data from observation notes, feedback sheet comments and video recordings of the sessions.

In relation to outcome components, the aim was to evaluate the impact of the programme in relation to pre-identified areas. These relate to research questions 4, 5 and 6 in section 1.5 of this report. Parallel versions of the parent, student and teacher transition questionnaires were used for pre-post programme comparison.

2.2.3.1 Transition programme evaluation questionnaire

The transition programme evaluation questionnaire (Appendix 19) was designed to provide a retrospective assessment of different components of the programme from the students’ perspective. The design and content of questions 1 to 4 was based on the Arfur Moe transition workbook evaluation

(OSSME, 2003a). All of the questions were closed (apart from a final catch- all question “do you want to add anything else?”). The first three questions were of the fixed yes/no response type. Question 4 comprised twelve four- point rating scale items presented in tabular format. 1 was ‘didn’t help me; 2 was ‘quite helpful’ and 3 was ‘very helpful’. A final ‘not sure’ category was added. Questions 5 to 13 focused on other elements of the programme. They utilised the same four-point rating scale.

The questionnaire was completed by the students in the latter half of the final session of the programme. Adults were able to provide assistance, if required, with reading and understanding of the questions.

Data was mainly quantitative in nature. Descriptive statistical analysis was undertaken using SPSS. The results are summarized in section 3.2.

2.2.3.2 Observation of programme sessions

The study incorporated direct observation of the sessions. Observation can be used as a “supplementary method to collect data that may complement or set in perspective data obtained by other means” (Robson, 2002, p.312). In this evaluation research, observation data were used to supplement questionnaire data and allow for exploration of the operation of the programme from the students’ and the researcher’s perspectives.

Robson (2002) refers to two main dimensions in his classification of observational methods. The first concerns the formality and structure

imposed on the observation. In this research, an informal approach was adopted. The envisaged advantage of this approach was that it would capture the complexity of the events and, in the case of the videotape data, would provide a more complete account of the situation. The second dimension relates to the extent of observer participation in the situation. This can range from complete immersion to what he refers to as “the ‘pure’ observer, seeking to be an unnoticed part of the wallpaper” (Robson, 2002, p.313). It was decided to adopt, as far as possible, a non-participative role. It was envisaged that this would enable full attention to be focused on observation of the operation of the sessions and the participants’ responses during different activities. Given that the researcher’s role was known to the participants, it is questionable whether you could consider this as non- participative in the purist sense. Furthermore, the researcher did provide comments for the feedback sheets thus participating in the operation of the programme.

It was decided to videotape all the sessions using camcorders mounted on tripods. The envisaged advantage of this method compared to sole reliance on field notes was that it would provide not only a permanent record but one that was not subject to potential observer bias and recording inaccuracies. In addition, it would provide both auditory and visual data, capturing the complexities of social interactions in a classroom context. Some of the disadvantages of using this technique were recognized, including its lack of flexibility compared with a human observer (Simpson & Tuson, 1995). For example, the students could block the camera or key events could be missed

if they were outwith the range of the camera. In an attempt to circumvent this problem, two cameras were employed for large group activities. At times it was necessary to have a camera in each room, for example, when the group was split for different activities. It was recognized that the presence of a camera could create an artificial atmosphere. To counter this argument, it was noted that after the initial novelty, the students came to accept the presence of the camera(s).

In addition to the utilization of video recordings for later analysis, immediate field notes proved helpful in providing an easily accessible record of events. Field notes were adopted as a recording method with two main purposes. Firstly, they would provide a source of information for the feedback comments completed at the end of each session. Secondly, they would inform the evaluative and forward planning discussions of the programme team at the end of each session. To this end, the author made brief notes of the operation of the activities in the six sessions using a prepared sheet (Appendix 21). This writer recognizes that one of the disadvantages of this method is its subjective nature. However, one of the advantages is its flexibility thus avoiding the potential difficulties associated with pre-set categories. In addition, both the author and the SALT made brief comments on the observable behaviour of the student participants (six and three sessions respectively) using a prepared sheet (Appendix 22). However, for the purposes of this report, only the notes made by the author will be subject to analysis. The reasons for this decision were that the SALT’s notes were not available for all the sessions; and that, in relation to research question

two, the focus was on the researcher’s perception of the implementation of the programme.

In analyzing the observation data, general comments about the operation of the activities and notes about the responses of individual participants were collated. This resulted in descriptive information about the nature of the activity and the level of student engagement (Appendix 23). Based on a number of readings of the data, the author generated a three-point rating scale designed to gauge the level of student engagement in each activity. A dictionary definition of “engage” is “to involve somebody in an activity, or become involved or take part in an activity” (Bloomsbury, 2001, p. 474). The term “engagement” is utilised in this analysis to encapsulate the level of attention and interest displayed by the students. The rating scale employed was:

1 = low (most of the students disengaged)

2= medium (approximately half of the students engaged) 3= high (most of the students engaged)

The results of the analysis are summarized in section 3.3.

2.2.3.3 Transition programme questionnaire

The decision to utilise questionnaire methodology to capture parents’, students’ and teachers’ views was based mainly on time constraints. Given the timescale for the research, it would not have been possible to conduct in- depth interviews with all participants prior to commencement of the programme. The use of parallel versions of the parent, teacher and student

questionnaire was designed to allow comparison across the three groups and provided triangulation using different data sources (Robson, 2002).

Pilot versions of the parent, student and teacher transition questionnaire were developed. The parents of two first year students in the autism unit were asked to complete the pilot version of the questionnaire and provide feedback. Following this consultation process, minor amendments were made to the layout of the table in question 6 (Appendix 9). Two first year pupils who attended the secondary autism unit were asked to complete the pilot version of the student questionnaire. In response to feedback comments, changes were made to the wording of questions 1 and 5; one of the items in question 6; and the end points of the two 6-point scale items in question 4 were altered to make ‘strongly agree’ point 1 in both (Appendix 11). The pilot version of the teacher questionnaire was very similar in content and format to the parent version. The principal teacher of the secondary autism unit asked two members of staff in the unit to review the questionnaire. Following this process, the only required amendment was to the layout of the table in question 6 (Appendix 15).

The items in question 6 were designed to assess the students’ skills in social interaction, social reasoning, social communication and friendship. In the student questionnaire, these were referred to as ‘social skills’. The aim was to develop a measurement instrument which would focus on specific behavioural skills and allow a pre-post programme comparison of observable social skills. Ideally the instrument would have three parallel versions,

namely student, parent and teacher. In addition, it would be suitable for use with students with ASD.

A number of potential measurement instruments were reviewed. The social skills questionnaires developed by Spence (1995) appeared to meet some of these criteria. Firstly, they were designed to measure “specific behavioural responses” resulting in “positive outcomes from social interaction” (Spence, 1995, p.1). Secondly, there were parallel parent, teacher and pupil versions. However, there was no research evidence as to the sensitivity of these instruments in detecting changes over a short period. This was important given that the six sessions of the programme were over a five-week time period. Spence (1995) acknowledges that information on the test-retest reliability of the instruments was not available at the time of their development and commented on the need to ascertain their sensitivity “to change following training or intervention programmes”. Another factor was the suitability of the instruments for students with Asperger Syndrome. This was an unknown element. Spence (1995) recognizes this aspect noting the value in obtaining information about the sensitivity of the questionnaires in “demonstrating differences in scores across groups of youngsters” (Spence, 1995, p.12).

After careful consideration of all these aspects, it was decided to produce an instrument which would incorporate skills applicable to young people with Asperger Syndrome and which would, hopefully, be sensitive to changes over a short time period. It should be noted that it was decided to utilise an

adapted version of the social skills questionnaire-parents (Spence, 1995) to measure students’ social skills as part of a longer term follow-up study of student progress following the transition to secondary school. The results of this research will be reported separately.

The twenty-six items in question 6 used a 6-point Likert-type rating scale ranging from 1 ‘very poor’ to 6 ‘very good’ in the student version and 1 ‘incompetent’ to 6 ‘competent’ in the parent and teacher versions. Statements were drawn from an observation profile which itemizes the types of behaviour patterns characteristic of children and young people with Asperger Syndrome (Cumine, Leach, & Stevenson, 1998). This profile has been developed for clinical use based on the ‘triad of impairments’ (Wing, 1992). For ease of completion, it was decided to make some amendments to the observation profile. This included a reduction in the number of items; changes to the wording of some statements; the inclusion of a few additional items (e.g. “understanding of personal space”); and changing from a 5-point scale to a 6- point scale. These amendments drew on the professional knowledge of the author in close collaboration with the principal teacher (AU) and speech and language therapist, all of whom have considerable knowledge and experience of working with children and young people with ASD and their families. The final version of question 6 contains twenty-six items; eleven focusing on social interaction skills; eight on social communication skills; and seven on social imagination and flexible thinking.

The post-questionnaires incorporated two changes. The two questions on personal passports were omitted. Analysis of responses to these questions in the pre-questionnaires suggested that respondents were unfamiliar with personal passports. That factor and time constraints influenced the decision to exclude personal passports from the programme content. The question which focused on the three things which would most help the student with the transfer from primary to secondary education was replaced by one which requested information on the three things that they would want staff in the secondary school to know about the student.

The method of distribution and completion of the pre-questionnaires was outlined in section 2.2.1.5. Post-questionnaires were completed by the majority of students and parents at a meeting held in the autism unit base in late June 2005. Follow-up home visits were arranged for those who had been unable to attend. In the case of one student and his parent this proved unsuccessful and a follow-up letter failed to elicit a response. The post- questionnaires for class teachers were distributed mid-June 2005 and returned using a stamped addressed envelope.

3 Results