An Evaluation of the LOVEBiTES and Respectful Relationships programs in a Sydney School (2012) was conducted by Michael Flood and Vicki Kendrick from the University of Wollongong in partnership with the National Association for the Protection of Children from Abuse and Neglect (NAPCAN). The LOVEBiTES and Respectful Relationships programs are run by a number of services in schools across NSW and, since 2010, also in a number of other states and territories. The evaluation seeks to understand the impact of the LOVEBiTES and Respectful Relationships programs had when conducted, with Year 10 and Year 7 students respectively, at one particular Sydney school.
The evaluation applied a pre and post-test design. That is, students involved in each program were tested before and after participating in the program and each student’s responses were matched and compared to assess whether their individual attitudes and self-reported skills changed over the course of the program. Although a follow-up element was initially planned, and the evaluation report notes this would have provided important additional insights about whether changes in skills and attitudes had persisted over time, this aspect of the analysis was abandoned due to difficulties with the data collection.
This evaluation is an example of an approach that conforms to a number of minimum standards of program evaluation, many of which are described in this chapter. In particular, it uses a model with three key elements: measuring the impacts of the program on social factors associated with effective violence prevention (e.g., knowledge, skills, attitudes and behaviours); using standardised measures to understand those impacts; and using a pre and post-program test design.
The evaluation also illustrates some of the challenges of school-based evaluations, particularly where data collection is attempted over a longer period of time.
9.2.3
Step 3: Identifying the information needed to answer the
evaluation questions and collecting the information
This step involves identifying how to collect the information (data) that will enable the evaluation questions to be answered. There is a range of types and sources of data and the availability of these will depend on the nature of the program and context in which it is being implemented. Where multiple programs are being evaluated or a program is being
implemented in more than one location, a common “outcomes framework” and/or consistently agreed outcomes indicators and measurement tools can help integrate findings across programs and locations. A rigorous evaluation design will involve more than one, in some cases, several types of data being collected and analysed. In relation to programs for 0–8 years, the question of how data should be collected raises some complex issues. In this context, data collection can involve professionals, parents and the children themselves, depending on their age. Data on children may be collected indirectly or directly. Indirect methods are based on observation or collecting information from parents and carers about their observations of children, using standardised and validated measures (see e.g., Bunston et al., 2013). The measures applied should be closely tied to the theoretical basis of the program. There are many different measures of child wellbeing, development and social functioning available. Direct data collection from children, particularly in the younger age groups, requires very careful consideration, both ethically and in terms of what is feasible from a cognitive perspective. The following resources are useful in considering these issues:
Sanson, A., Misson, S., Hawkins, M., and Bethelsen, D. (2010). The development and validation of Australian indices of child development part 1: Conceptualisation and development. Child Indicators Research, 3(3), 2010, 275–292.
Sanson, A., Misson, S., Hawkins, M., and Bethelsen, D. (2010). The development and validation of Australian indices of child development part 2. Child Indicators Research, 3(3), 293–312.
Barblett, L., and Maloney C. (2010). Complexities of assessing social and emotional competence and wellbeing in young children. Australasian Journal of Early Childhood, 35(2) 13–18.
Moore, T., McArthur, M., & Noble-Carr, D. (2008). Taking little steps: Research with children—a case study. In Australian Research Alliance for Children and Youth (ARACY) & NSW Commissioner for Children and Young People (NSWCCYP) (Eds.). Research with Children and Young People: A Compendium. Sydney: ARACY & NSWCCYP. <telethonkids.org.au/media/54379/involvingchildrenandyoungpeopleinresearch_1_.pd f>.
The main types of data are now described.
Administrative data from the program/organisation covering issues such as: the number of clients who completed the program or to whom services were provided and information about referrals in or out of the program. Programs are often required to maintain these kinds of records for the purpose of reporting to funding bodies.
Quantitative data from surveys. These data support statistical assessment across a range of areas. These may include experiences (e.g., questions about whether the program worked for you), and attitudes (e.g., are particular kinds of attitudes more or less common after the program). Surveys involve information being collected in a format where the questions are carefully worded and participants are required to choose an answer from a pre-determined series of possible responses. Surveys may be administered by pen and paper, over the telephone or online. The decision about whether to use this approach, and in which format, should be based on the number of potential participants that may be surveyed and their levels of literacy and access to computers and telephones. The decisions
that need to be made in this area can be quite complex, but they may also be limited by the strategies available for practical or resource reasons. These kinds of data should be collected from staff and clients. Consideration should be given to including other stakeholders in data collection, such as referring agencies, as this can contribute valuable insight into how a program is operating.
Qualitative data is information based on in-depth interviews or focus groups with professionals associated with the program and clients. Unlike quantitative data, information collected in this way does not lend itself to statistical analysis and comparison. However, it allows rich information to emerge that can be very informative, particularly in understanding the perspectives and experiences of individuals and developing an understanding of how a program works. An interview format based on open-ended questions allows issues to be explored flexibly and deeply. Focus groups support a dynamic and interactive discussion that can be particularly useful for exploring practitioner perspectives. Caution should be used in applying focus groups for client data collection due to the possibility that sensitive material may be disclosed and confidentiality is not in the interviewer’s control in a group setting.
Principles and implications
Where multiple programs or sites are being evaluated, a common outcomes framework and agreed outcomes indicators will support the development of integrated findings. Administrative data sources should be sought for any evaluation. For new programs,
program design and reporting requirements can be established to provide useful administrative data for evaluation and reporting purposes. Such data may include inward referral sources, outward referral patterns, client commencement and completion data, commencement and completion timeframes, and client demographic data.
For most programs, optimum evaluation designs include quantitative and qualitative data sources, unless the target groups for programs are too small and too diverse to support quantitative approaches. Data sources should include staff, parents and children and may include other stakeholders. In some instances, where the reach of a program is intended to be widespread and attitudinal, such as school-based approaches, an evaluation based on quantitative data from the target audience may be a justifiable primary strategy supplemented by qualitative insights from program staff and clients. Data collection methods need to be carefully considered to accommodate differences in
language, literacy and levels of cognitive development.
Issues related to sensitivity and confidentiality should also inform choice of data collection methods. Focus groups may not be appropriate in some instances. Interviewers and interview approaches need to be carefully considered to support appropriate responses to sensitive issues. In some instances, the gender of the interviewer may need to be considered. Interviewers should be trained to be sensitive to the needs of the particular client group. This may include cultural sensitivity.