Developing Indicators at Different Levels and for Different Stages

Essentially, for DFID, we can envisage three different sets of indicators: those at the sectoral level; those at the planning and pre-planning stage; and those which would be used for monitoring and evaluation.

Sectoral Level

If the focus is the sector, (and reflecting on the OECD experience) there is only a limited choice given the types of data that can be collected. The main problems are the mechanisms of collecting the data and its quality. Technology for carrying out sample surveys, however, is now well developed and, with country agreement, this is probably the best way forward.

Here the second axis of the table could be either the level or mode of decision-making. We have chosen to take the level as the crucial dimension, as this usually determines the nature of possible participation. Here the entries in the cells are more in the nature of values relative to presumed targets for the sector (without falling into the trap of setting targets which will lead to falsification of data).

The table would have to be differentiated according to whether the focus of performance indicators was:

• the formal or informal system (CONTEXT);

• qualitative improvement or equity of access (AIMS);

• types of INPUTS and preferred PROCESSES;

• nature and timing of OUTPUTS and OUTCOMES.

This would, therefore, involve at least four tables according to whether the formal or non-formal system was being considered and whether the overall aim was qualitative improvement or equity of access. The variation between types of inputs and processes, and between kinds of outputs and outcomes (the latter closely connected with the AIMS) may not, however, be sufficient to generate substantially different tables. Table 3: Planning Process and the Need for Performance Indicators at Different Levels

Context Aims Inputs Processes Outputs Outcomes

For the non-formal system, the distinction made above in section 1.5 between different types of non-formal education and the corresponding types of performance indicators have to be taken into account as well as the distinctions made in Table 3. Lockheed and Levin (1991) suggest that context should be specified in terms of facilitating conditions such as community involvement, school-based professionalism, flexibility, and the will to act as reflected in vision and decentralised solutions. They do not specify exactly how one is meant to measure any of those and only seems feasible at the most local level. Their list of inputs, however, is probably as good a starting point as any:

curriculum, instructional materials, time for learning, teaching practices (Lockheed and Levin, 1991).

Planning and Pre-Planning of Projects

The approach here will probably have to be rather different. Consider the list cited in section 1.5.2.(2) above for the planning of projects:

What is the evidence of demand for the education service proposed? What will be the benefits to the country and individual? Can these improvements be measured and where possible quantified and qualified? Will there be cost economies resulting? Is the

proposed strategy seen as the most cost effective? Are the recurrent cost implications manageable?

Recommendations: Each of these questions is likely to generate its own set of

indicators - almost certainly too many for consistent judgements to be made - and each might well require a different set of data which would be costly to collect. Inasmuch as the answers to these questions are agreed to be important criteria for choosing projects, then perhaps the best approach is to propose what detailed specifications of indicators would have looked like in respect of a number of projects - some funded, some not funded - in order to assess whether or not those criteria are actually taken seriously, in deciding upon funding.

If we focus on the project, then it is appropriate to consider project-specific

performance indicators (PSPIs). These are, in principle, straightforward measures of the extent to which a project completes the defined tasks: in the terminology of the logical framework, these are the 'purpose-level' indicators.

The issue is whether there is any scope for consistency in terms of the sets of

performance indicators used in the logical frameworks for different types of projects (similar to the approach adopted by the Health and Population Division). This should be feasible for a large proportion of projects but the main problem is to assess exactly how these are used. At worst they may generate perverse incentives (as explained) or they may simply be ignored. In all cases, the importance of involving the 'beneficiaries' is crucial.

If we are considering a project then we need to be able to develop indicators for the context, aims, inputs, processes, outputs and outcomes (the same kind of list as in the logical framework) at different stages of the project cycle: distinguishing (at least) between pre-planning, start-up, mid-term evaluation and follow up. On this basis, an appropriate framework could be as follows:

Table 4: Performance Indicators at Different Stages of the Planning Process and of the Project Cycle

Preplanning Startup Mid-term Evaluation Follow-up Context Aims Inputs Processes Outputs Outcomes

Essentially, the approach here is an extension of the project framework methodology which, although it is likely to miss crucial process characteristics, is used to assess the whole project process from start-up through to evaluation. That methodology was never intended to be used at the pre-planning phase; and although it is recommended for use as a basis for follow-up evaluation, the context may have changed dramatically, so that it may not be appropriate. Moreover, not all of the basic components would be

appropriate at each stage.

In any practical application, of course, one would need to explode each of the cells, in terms of specifying the level at which indicators are required and, eventually, the extent

of participation in the process.