2.5 E VALUATION F RAMEWORK
2.5.3 Types of evaluation frameworks
2.5.3.1 Sustainability framework
The inception of the sustainability framework, which originated in Sydney, Australia, arose from the need to develop a methodology for evaluating the overall sustainability of urban water systems (Lundie et al., 2005). This methodology was being used to assist the Australian urban water industry to achieve more sustainable use of scarce water resources. Through this methodology, the industry would evaluate the sustainability of the various supply and demand options while considering the economic, environmental, human health, technical and social factors (Lundie et al., 2005).
The development of this methodology was further spurred on by the Intergovernmental Agreement for a National Water Initiative in Australia which identified the need to create water-sensitive cities. This initiative especially called for national guidelines for evaluating options for
Mburu Mwangi
The feasibility of implementing advanced metering technology in high income areas in SA 39 water-sensitive urban developments in both new urban sub-divisions and high-rise buildings (Lundie et al., 2005).
In the development of the framework, stakeholder involvement and iterative procedures of activities were identified as vital and critical attributes were needed to establish more sustainable options for the water industry. As such, the proposed layout of the framework consisted of six phases as is illustrated by Figure 2-21. Each phase consisted of two components, the first addressing the procedure of how to carry out each phase and the second component focusing on the need for stakeholder participation. Figure 2-21 displays the final proposed layout, illustrating the different phases and the procedure of each phase.
Figure 2-21: Layout and phases of a Sustainability Framework
Sustainability framework can be made flexible to allow its application for systems varying in the following:
• Scale of participation necessary
• Financial requirements
• System boundaries of a project
Mburu Mwangi
The feasibility of implementing advanced metering technology in high income areas in SA 40
• Asset life-time (Lundie et al., 2005)
However, despite its flexibility in the above stated areas, the sustainable evaluation frameworks procedure and protocols that govern the evaluation process are rigid.
2.5.3.2 Composite Indicators Framework
A Composite Indicator Framework, also known as a Composite Indicator, is an aggregate of all dimensions, objectives, individual indicators and variables used (Nardo et al. 2005 and 2008). It is formed when individual indicators are combined into a single index on the basis of an underlying model of the multi-dimensional concept that is being measured (OECD, 2004).
Through the Composite Indicator Framework, multi-dimensional concepts (such as competitiveness or e-trade) which cannot be captured by a single indicator can be measured (OECD, 2004).
Composite Indicators (CI) are often used wherever a plurality of variables are needed for evaluation and comparative purposes (Munda & Nardo, 2005; Nardo et al., 2008). Composite Indicators have been widely utilized to compare the performances of countries in fields such as the environment and technical development, and they are recognized to be useful tools in policy analysis and public communication. This is due to the simplistic manner in which Composite Indicators outputs can be compared and hence interpreted as compared to finding trends with separate indicators (Nardo et al. 2005 and 2008).
However, Composite Indicators can be misleading if poorly constructed. Therefore, to avoid misinterpretation, both a well-developed framework and good quality data are vital (Nardo et al., 2005, 2008). To construct an adequate Composite Indicator Framework, a sequence of steps needs to be adhered to as is illustrated in Table 2-1.
Table 2-1: Description of the different steps of the construction of Composite Framework (adapted from Nardo et al. 2005 and 2008)
Design Step Description
1 Theoretical framework
In the first phase of constructing a Composite Indicator, a theoretical framework must be developed. The theoretical framework is developed to provide the basis for the selection and combination of single indicators. With regards to advance metering, the selected indicators will be gauging the economic vitality, environmental friendliness, social acceptance, and technical practicability of both conventional and advanced meters.
2 Variable selection
Mburu Mwangi
The feasibility of implementing advanced metering technology in high income areas in SA 41
In this phase, indicators are selected based on their analytical soundness, measurability and relevance to the phenomenon being measured and relationship to each other. For this project, the clarification of how the economic vitality, environmental friendliness, social acceptance, and technical practicability performance will be accessed should be well elaborated and understood in order to collect the required data.
3 Imputation of missing data
In this phase, careful consideration is given to various approaches that can be taken for imputing missing values. Extreme values should be carefully examined as they unintended benchmarks.
In this phase, indicators are normalised to render them comparable. Attention needs to be paid to extreme values as they may influence subsequent steps in the process of building a composite indicator. Skewed data should also be identified and accounted for.
6 Weighting and aggregation
In this phase, indicators are aggregated and weighted according to the underlying theoretical framework. Correlation and compensability issues among indicators need to considered and either be corrected for or treated as features of the phenomenon that need to retained in the analysis.
7 Robustness and sensitivity
In this phase, an analysis needs to be undertaken to assess the robustness of the composite indicator in terms of, for example, the mechanism for including or excluding single indicators, the normalisation scheme, the imputation of missing data, the choice of weights and the aggregation method.
8 Presentation and visualisation
In this phase, composite indicators are visualised or presented in a number of different ways which can influence their interpretation.
Mburu Mwangi
The feasibility of implementing advanced metering technology in high income areas in SA 42 2.5.3.3 Results Framework
The success or failure of an implemented project is difficult to gauge unless the expected results are clearly articulated. Based on this premise, results-based management is utilized as a key tool to monitor development effectiveness (World Bank 2012).
A Results Framework is an example of a tool that can be utilized for results-based management. A Results Framework is an explicit articulation (graphic display, summary, or matrix) of the different levels of results expected from a particular project (World Bank 2012).
The Results Framework illustrates the direct relationships between the intermediate results of activities all the way to the overall objectives and goals. It also outlines how each intermediate results/outputs and outcomes relate to and facilitate the achievement of each objective as well as how objectives relate to each other and to the main goal (UN-Women 2011). Figure 2-22 displays an example of an implemented Results Framework.
Figure 2-22: An example of an implemented Results Framework for an awareness-raising campaign around domestic violence legislation (adapted from UN-Women 2011)
2.5.3.4 Logical Framework
The Logical Framework, also known as Logical Model or Project Planning Matrix, was developed for the United States Agency for International Development (USAID) as a tool to aid
Mburu Mwangi
The feasibility of implementing advanced metering technology in high income areas in SA 43 in the conceptualization and strengthening of a project design, the implementation and evaluation and the analysis of the assumptions behind it (Gra, 1990; Bakewell & Garbutt, 2005; UN-Women, 2011). Since the development of the Logical Framework, it has been adopted, through various revisions, by a large number of bilateral and international development organizations.
The Logical Framework has proven extremely valuable for project design, implementation, monitoring, and evaluation (Bakewell & Garbutt, 2005).
The Logical Framework provides a linear, “logical” interpretation of the relationship between inputs, activities, outputs, outcomes and impacts with respect to objectives and goals. It outlines the specific inputs needed to carry out the activities or processes to produce specific outputs which will result in specific outcomes and impacts. Logical Frameworks form the basis for monitoring and evaluation activities for all stages of a system (UN-Women, 2011).
Logical Frameworks are valuable tools for:
• Programme Planning and Development: the Logical Framework structure aids with the programme strategy, i.e. to help clarify what stage the programme is at and where it should be.
• Programme Management: through "connecting the dots" between resources, activities, and outcomes, a logical model can be used as the basis for developing a more detailed management plan. Using data collection and an evaluation plan, the Logical Framework helps to track and monitor operations to better manage results. It can serve as the foundation for creating budgets and work plans.
• Communication: a well-built Logical Framework is a powerful communications tool. It can show stakeholders at a glance what a programme is doing (activities) and what it is achieving (outcomes), emphasizing the link between the two.
Logical Frameworks are presented as diagrams connecting programme inputs to processes, outputs, outcome and impact as they relate to a specific problem or situation. A Logical Framework shows what resources the programme will need to accomplish its goals, what the programme will do, and what it hopes to achieve, emphasizing links between these aspects.
Figure 2-23 illustrates the typical format of the Logical Framework.
Mburu Mwangi
The feasibility of implementing advanced metering technology in high income areas in SA 44
Figure 2-23: A typical layout of a Logical Framework
Mburu Mwangi
State of the art in advanced metering technology in high income areas 45
3 CASE STUDIES
3.1 Introduction
Case studies are recorded reports on projects and schemes that have been previously implemented to solve a certain problem or improve on a current situation. Case studies give a summarized account of the situation prior to the project, the reason for implementing the project, the implementation process as well as the output of the project.
This chapter discusses case studies found in the consulted literature. They serve to illustrate scenarios in which advanced metering was the preferred solution and illustrate the objectives these meters were to met, These case studies were sourced from manufacturing companies, projects (both completed and on-going) and other sources.