The evaluation procedures

The essential purpose of evaluation in the context of collaborative planning processes is to identify whether an initiative is likely to be successful in accomplishing its objectives, and continually reviewing the process to make incremental improvements throughout to try to ensure that targets will be met (Drafting Group 2002; Kelly, Essex & Glegg 2012). Evaluation is a powerful learning tool to generate lessons for future practice and to satisfy the greatest number of stakeholders, thereby enhancing their commitment to plan implementation (Lengrand & Associés 2006; Morton 2009). Evaluation should not be an afterthought but it should be built into the design of the process from the beginning. Moreover, evaluation needs to be a cyclical process questioning the legitimacy of participatory processes and the appropriateness of the solutions they develop (Innes & Booher 1999). The continual evaluation of a collaborative process will help the conveners to determine if some areas need to be refocused to ensure that the process is moving towards the intended outcomes (Gepraegs 2008). The main types of evaluation required to set up a proper framework for evaluating the collaborative planning process to improve tangible and intangible outcomes will now be discussed.

Evaluation can generally be categorised based on its purpose into three main types: i) Accountability evaluation aims to make some judgement on the outcome qualities. It generally compares the results against the initiative’s objectives or a set of criteria which were developed by the evaluators. It is the most common evaluation form; ii) Formative evaluation aims to provide information to improve stakeholder participation through evaluating the outcomes. There are few case studies in the context of the stakeholder participation that have used and discussed formative evaluation in the literature (such as regional planning for South East Queensland) (Cameron & Johnson 2004); and iii) Knowledge evaluation seeks to contribute to an understanding of the factors which affect stakeholder participation in the process and activities. This type of evaluation requires some theoretical framework to explain and compare the results (Cameron, Smith & Johnson 2005). According to many, including Innes and Booher (1999) and Kelly, Essex and Glegg (2012), a systematic evaluation to assess both the process and outcomes will be more effective because the two cannot

93

be neatly separated in a consensus building process. It seeks to examine process issues such as whether all relevant stakeholders were included, or whether the participation activities were appropriate. It also focuses on assessing the tangible (plan, agreement and regulations) and intangible (social, intellectual and political capital) outcomes, such as whether a high-quality agreement was achieved or whether shared intellectual capital was built.

3.4.1 Evaluation framework

The evaluation framework is grounded in a systematic evaluation, which is considered the most proper type for the objectives of this research because it assesses the success of collaborative planning based on a multiple criteria evaluation for both the process and the outcomes. The main principles of this framework will be: i) the evaluation should be based on clear criteria covering the spectrum of the potential benefits from the collaborative process and outcomes (Innes & Booher 1999); ii) all stakeholders should be involved in the evaluation procedures; iii) an external evaluator is required for an unbiased evaluation and is able to identify actual weakness and issues (OECD 2004); and iv) the evaluation needs to be continual and carried out at different times through a project’s lifetime (Gepraegs 2008).

Although there is a rich body of literature discussing evaluation from two perspectives to cover the process and the outcomes, a third layer needs to be added to the evaluation framework to cover the pre-process stages of collaborative planning which potentially widens the benefits of stakeholder collaboration and increases the likelihood of the plan being implemented. Therefore, this evaluation framework comprises three main layers with each layer consisting of two main components (Table 3-4). The first layer considers pre-process evaluation to assess the previous studies and attempts to evaluate the quality of the information as a basis for the planning process and decision-making. The second layer is the process evaluation, which seeks to assess both the quality of the process and any adjustment made to remedy identified problems before the implementation stage. The last layer is outcome evaluation taking both an intermediate and long term perspective.

94

Table (3-4) The evaluation framework

Evaluation layers Concept Objective

P re -pr oc ess Evaluating the previous studies and attempts

To avoid their problems and maximise the

potential through the new initiatives

To determine actual reasons why previous plans were not implemented

Input evaluation (inputs into planning process)

Improving the

information basis for the planning process will improve the outcomes and lead to better results

Input indicators measure the quality of the information, data, and analysis that goes into the planning process before embarking on any decision

P roc ess Process evaluation (quality of the process)

Improving the process of decision-making to ensure more satisfactory outcomes and remedying the process problems

Process indicators provide benchmarks for evaluating the planning process and steps to adjust its problems and improve the outcomes

Output evaluation (assesses the quality of the plan or agreement) *

Improving outputs will ensure more success in producing results

Output indicators measure the products of the planning process, such as plans, policies, and regulations before the

implementation stage to lead immediate plan improvement and changes The outcome s Performance indicators (measures of tangible and intangible outcomes’ effectiveness)

Improve the outcomes through shorter feedback loop

Performance indicators (also called intermediate outcomes) assess the shorter-term performance of specific plans. For example, a quarterly monitoring report prepared to determine whether to continue or change the implementation strategies Outcome evaluation

(impact measures)

Monitoring of actual results is important to measure success of the efforts

Outcome measures (also called long- term outcomes or impact measures) to assess the actual on the ground changes that result from the combination of plans, and social, economic, and environmental trends

Source: The author based on Cameron and Johnson (2004); and Margerum (2011)

*output evaluation aims to improve and adjust the outputs before the implementation stages so the framework puts this component within the process evaluation layer

The evaluation framework needs to be taken into account when designing any collaborative planning process as it seeks to explicitly define objectives (preferably quantified), combined with timetables for achieving objectives and benchmarks against which progress and outcomes can be assessed. This evaluation framework should be iterative and seek to improve the process and outcomes as well as to make

95

the collaborative planning process more transparent for the participants (Drafting Group Cameron, Smith & Johnson 2005; 2002).

Consequently, the research questions are:

- Was there any evaluation procedure for these initiatives? If yes, when and by whom were the initiatives evaluated?