Phase 4: Evaluation Phase

�����

Refer to Checklist 9, Annex E for detailed requirements of the creative phase.

2.4.5.6  Phase 4: Evaluation Phase

The objective of the evaluation phase of the VE job plan is to analyze the results of the creativity phase and, through review of the various alternatives, select the best ideas for further expansion.

Evaluation Phase Outline:

� Perform preliminary screening to separate the best ideas.

� Evaluate the alternatives to aid development of solutions.

� Determine criteria and objectives.

� Weight the alternatives.

� Weight the criteria and objectives of the project.

� Compute the numerical rating.

� Rank the alternatives.

� Select the best alternates for development.

During creativity, the group makes a conscious effort to prohibit any judicial thinking so as not to inhibit the creative process. Now the ideas produced must be critically evaluated for acceptance.

Use the key questions listed below as the basis for a set of evaluation criteria to judge the ideas:

� How might the idea work?

� Can it be made to work?

� What is the cost?

� Will each idea perform the basic function?

� Which is the least expensive?

� Can it be modified or combined with another?

� What are the chances for implementation?

� Will it be relatively difficult or easy to make the change?

� Will the users' needs be satisfied?

� What is the savings potential, including life cycle costs?

2.4.5.6.1  Preliminary Screening Techniques

Several techniques are available to evaluate alternate ideas. Examples are as follows:

� Comparison Technique. This technique is a method of comparing the various features of all of the alternatives under consideration.

� Advantages Versus Disadvantages Technique. In this technique, list the advantages and disadvantages of each alternative. Then sort the ideas based on the number of advantages and disadvantages of each. Choose those ideas with the greatest total advantage for further evaluation.

� Ranking Technique. The ranking technique allows the evaluators to assign a numerical rating to the alternates. Start this process by judging an excellent idea to be worth 5 points; a good idea, 4 points; a fair idea, 3 points; a poor idea, 2 points; a very poor idea, 1 point. Group all 5-point ideas and evaluate

���������������������������

�����

further. Use the same method on all 4-point ideas, and, if necessary, on the 3-point ideas.

� Probabilities Technique. In this technique, assign subjective probabilities of success to the implementation of acceptable ideas. A probability close to one means that the idea has a good chance of being implemented. A probability close to zero means the idea has little, if any, chance of being adopted.

� Estimating Alternatives. Rank the remaining alternatives according to an estimate of their relative cost-avoidance potential. Base the ranking on relative estimates comparing each of the alternatives against the original design method for providing the function. Develop the surviving alternative further to obtain more detailed cost estimates. Proceed to more detailed cost estimates only if the preceding step indicates that the alternative is still a good candidate.

2.4.5.6.2  Weighting Criteria and Objectives

The method most commonly used for the evaluation process is the criteria scoring matrix. By comparing each criterion against the other and assigning a measure of importance.

Table 2-5 shows an example of a "weighting matrix" used in this method.

Measures of importance may be as follows:

4 = Very Major Preference 3 = Major Preference 2 = Minor Preference 1 = Slight Preference

In the case of ties, each criterion is assigned 1 point.

It is seldom that "objectives" or "criteria" will be of equal importance; some should have greater influence on the final decision than others. A weight factor or weight of importance can then be introduced. Weights from 1 to 10 are often used with the highest number given to the criteria with the most importance and the other raw scores are adjusted in proportion (rounded off).

Following the development of "Weights of Importance", an analysis matrix is used to analyze each alternative option being considered is initially judged on a 1 (Poor) to 5 (Excellent) basis and that score is recorded in the lower portion of the square. This number is then multiplied by the "Weight of Importance" number and recorded in the upper quartile of the square. Finally the sum of these numbers is recorded as a "Total".

Do not arbitrarily discard any idea; give a preliminary evaluation, as objectively as possible, of each idea to determine whether or not there is some way the idea can be made to work. These totals are then used as a tool to aid the team in arriving at the "Best" alternative but should not be considered as totally conclusive.

In this example an individual is considering the purchase of an automobile and wishes to utilize the "Matrix-Weighted Evaluation" process to objectively

���������������������������

�����

determine the best value of three automobiles being considered. In order to accomplish this, the individual identifies the criteria he/she wishes to consider and records them on the matrix sheet. For this example the criteria are: cost, appearance, comfort, performance, and safety.

Next, the individual weighs each criterion against the other. In this example the decision is that cost is better than appearance by a factor of "2" same for cost versus comfort; and cost is better than performance by a number of "1". This process is continued until each criterion is compared to all other criteria.

Next, each of the Letter scores is added and the sum recorded below as a "Raw Score" and then weighted as described above. Finally it is shown that Automobile A is favored by a score of 86 over B (80) and C (77).

The example of a Weighing Matrix uses information to illustrate the method. The actual information for a project should be used in the Weighting Matrix.

���������������������������

�����