Indicators of Implementation

An attempt was made to analyse the project concerns with planning and implementation in terms of five broad areas for action and to ask, in each case, three further questions. First, what processes would need to be set in place to ensure that this priority is

reached? Secondly, what process or quality indicators might be quickly and easily collected which might suggest that progress towards the priorities was being made? And third, by what methods, with what frequency and by which individuals and groups, might the relevant information be collected?

The main priorities were identified as follows:

• the extent to which capacity building for programme implementation had started and the stage that has been reached;

• the extent to which steps had been taken to help the local community participate in the affairs of the local school;

• the efforts made to enhance girls' enrolment and that of disadvantaged groups;

• the provision catering for the diversity of children's learning needs; and

• the steps taken to provide effective external support for teachers to enable them to perform more effectively.

The kinds of data required to assess project management included the following:

• nature of participation at different levels

• staffing of institutions

• physical infrastructure and any proposed improvements

• planning documents, site and supervision reports

• use made of non-personal resource inputs

These data were then used to construct the following indicators:

• the extent to which participation is seen as useful

• shortfalls of staffing compared to establishment, by programme functions

• levels of maintenance of non-personnel resources

• quality of plans

Indicators of Institutional Development Indicators of linkages at each institutional level

included:

• the areas of linkage (e.g. at the level of the State Centre for Educational Research and Training level, including training of Master Trainers,

evaluations and research studies, production of materials); and

• the extent and effectiveness of the linkage process.

Three aspects of institutions were evaluated: training, materials development and ongoing professional support.

Training

Data were collected on the number of training courses and clients. This included

subsets such as the training of Non-Formal Education/Alternative Education instructors, and in some areas, the training of Early Childhood Care and Education (ECCE)

teachers; the timing of training (e.g. state training programmes intended for cascade might be initiated at times when training lower down the levels will occur at

inappropriate times in the year); and processes of training.

The methods by which such data can be captured include survey and document analysis to pick up information on the number of modules offered, attendance numbers and timings of courses. The key indicators would be adequacy, organisation and relevance

of the training process, for example, coherent sequencing of the training process (i.e. that the production of the required materials is in place before the training modules are implemented, etc.)

Materials Development

Basic contextual data include: the number of modules developed; the process of module development; and the availability and use of modules.

Methods for collecting this data include document analysis, use of focus groups, etc., discussions with recipients of training, participant observation, and structured and semi- structured interview techniques.

Ongoing Professional Support

Basic background data included: the frequency, number and purpose of visits to the school; then number and kind of follow-up activities; and the types of ongoing dialogue with teachers.

Key indicators were based on whether there is two-way communication or whether directives and training are delivered and no feedback is received; or whether needs and concerns of clients (generally teachers) were heard and responded to by the institution concerned.

A suitable timeframe for the simultaneous conduct of the evaluations involved

collecting data, analyse and report back on the first component of the evaluation by the end of Year 2 of the project. This provided the appropriate stakeholders with some early information about the efficient implementation of training at a stage where corrective, formative and innovative interventions would be made. Capacity building and empowering of lower levels would be a guiding principle for data analysis. A further round of data collection and analysis will be made as a follow-up to subsequent interventions or ongoing implementation, sometime in the middle of the project. The exact decisions about when to initiate the follow-up study would be a state-wide

decision. A final review would be made at the end of the seven year life of the project, with a particular emphasis, perhaps, on examining the sustainability of the institutions' effectiveness and quality.

Indicators of Community Participation

It was seen as important to find a way of collecting data on the part played by the community in the collection of data about enrolment and repetition, for example, through child-to-child monitoring, door-stepping, adopting of specific families/groups

by the VEC - as an indicator of how much involvement there is at a real level from the VEC and other community members. The quality of community involvement in

education, however, also requires indicators that register the satisfaction level of teachers and parents with the functioning of the VEC.

On an institutional level the setting up and functioning of VECs, the records of meetings, the incidence of discussions, the occurrence and resolution of critical incidents, have all been measured and taken as indicators of community involvement. An indicator of growing co-operation and mutual support, for example, was the number of times that a teacher felt able to initiate questions or discussion with the VEC. Finally, a quality component was incorporated in the indicator itself: e.g. could members of the VEC discuss educational issues showing some understanding of the project aims in classroom processes?

Indicators of Access, Enrolment and Repetition

The collection of data through the school and pupil survey to supplement the initial baseline survey and Project (PMIS) and EMIS data gave an aggregated source of information at the level of numbers of children from the village who are school-going children, the numbers of marginalised group children attending/not attending school and the number of drop-outs in a year. More detailed data needed to be collected from the population, and for this, there were some village-based exercises such as censuses and registers. However, households from which students come could also be the basis of a sample of households who send at least one child of school age (6-14) to school. Indeed, it is almost a PPS (probability proportional to size) random sample of

households who send at least one child to school, but not quite because a household with, say 5 children may only be sending three of them to school. The correct weighting for the number of children in that age group whom they actually send to school,

however, can easily be calculated from data collected at the interview stage.

It will be important to study a sub-sample of this kind for a number of reasons linked to the equity, enrolment and community participation objectives:

• the reasons for sending some or all children to school;

• the encouragement given to, or motivation of children to go to school; and

• involvement in school (PTA, homework, etc.)

The school and pupil survey will yield quantitative, factual data which will be able to be used in investigations of community participation. The triangulation of figures through interview, small studies, Participatory Community Appraisal (PCA) and a

number of small case studies will add detail and reveal new and interesting questions, throw light on wider issues and corroborate data collected through the baseline survey and PMIS, EMIS. It will be possible to cross-reference across similar or different communities with similar project inputs, e.g. new school, trained teachers, and to note patterns of similarity or difference. This will provide the basis for further qualitative investigation.

Indicators of Teacher Training

From the point of view of the project, the quality of training can be thought of as evidenced in the changes that occur in teaching and teachers as a result of the training input. It is assumed that pedagogic and attitudinal changes in teachers can lead to improvements in children's learning, classroom interactions, and perhaps in terms of teacher attendance and retention also.

The in-service training of teachers can be divided into three sequential stages: preparation of materials, planning and implementing the programme, and the effectiveness of the programme in practice.

The Preparation of Materials Prior to Planning

Indicators of stakeholder participation in the preparation of materials proposed are:

• whether teachers have contributed towards the planning and preparation of materials;

• whether teachers have provided examples of good practice explicitly related to the training; and

• whether there have been trial runs of the training materials before planning, in order to smooth out any problems with the materials.

Organisation of the Training Programme

Assuming that basic data about the programme (for how many, how long, where it takes place, methods of delivery, content) has been collected, the kinds of indicators of effective functioning would be:

• whether actual training matches the plan and how the training programme is assessed;

• the extent of active involvement of participants including the existence of follow up sessions and their feedback; and

• the sustainability of the programme.

The Effectiveness of the Training Programme in Practice

The final outcome of training is its effect on classroom methodologies and children's learning. This will be considered in the sections on school processes and learning achievement. Specific training indicators relate to:

• whether teachers have changed their pedagogy or whether they mimic the model lesson when they are being observed and in all other lessons revert to didactic, front-of-class instruction;

• efficiency in terms of cost and the use of existing structures and trainers; and

• is there a significant transmission loss in the cascade model and does this vary between teachers and trainers?