intellectual product; their evaluation must be based on the individual professional judgment of compe- tent evaluators and should not be reduced to a purely arithmetical exercise. The difficulty is to ensure that this judgment is not exercised in an unreasonable or arbitrary manner. Evaluators may, either consciously or unconsciously, manipulate the points awarded to specific aspects in the evaluation for a number of rea- sons, including inadequate experience in the field of the assignment or in evaluating proposals of this nature. Therefore, it is important that subjectivity implicit to any individual professional judgment be comple- mented by transparency, consistency, and fairness. The individual evaluator entrusted with evaluation should, when required, be able to explain to the sat- isfaction of a qualified reviewer from the higher authority, the public, or the Bank the reason for his or her recommendation.
One way of helping to achieve the above is by adopting a suitable rating system for the evaluation of technical proposals under the criteria and subcriteria (if provided) established in the RFP.
This chapter complements chapter 16 because it provides detailed practical recommendations for rat- ing evaluation criteria (and subcriteria) and scoring various parts of the technical proposal. (Appendix 7 of this Manual provides examples of complex technical evaluations.)
17.1 Rating System
The World Bank Standard RFP (SRFP) specifies the criteria used to evaluate technical proposals and the points (or weights) given to each. The responsiveness of a proposal to the TOR is determined by its respon- siveness to the adopted evaluation criteria (and sub- criteria) indicated in the Data Sheet of the RFP.
C H A P T E R
17
Evaluation Practices of
Technical Proposals
For the Full Technical Proposal (FTP), the crite- ria include the following:
•
Specific experience of the consultants relevant to the assignment•
Adequacy of the proposed methodology and work plan in responding to the TOR•
Key professional staff qualifications and compe- tence for the assignment•
Suitability of the transfer-of-knowledge (capacity- building) program•
Participation by national consultants among pro- posed key staffThe “transfer-of-knowledge” (capacity-building) criterion is included wherever it forms an explicit as- pect of the assignment. The “participation by national consultants” criterion is optional for the Borrower.
For the Simplified Technical Proposal (STP), the criteria include the following:
•
Adequacy of the proposed methodology and work plan in responding to the TOR•
Key professional staff qualifications and compe- tence for the assignmentFor both the FTP and STP, the RFP should spec- ify the subcriteria for the “key professional staff quali- fications and competence,” as indicated in the SRFP. For the FTP, the RFP should also indicate subcriteria for the “adequacy of the proposed methodology and work plan,” and those, if any, adopted for the other cri- teria. For all criteria and subcriteria, the RFP should al- ways indicate the points to be allocated to each of them.
In the RFP, the points assigned to a particular cri- terion (or subcriterion) indicate the maximum score (maximum number of points) that can be allocated to it. The actual score given by the evaluator indicates the degree to which the proposal being evaluated under that particular criterion (or subcriterion) meets the requirements (that is, its level of responsiveness). The
90 EVALUATION PRACTICES OF TECHNICAL PROPOSALS Rating System
1 7 . 1
level of responsiveness for each criterion (and subcri- terion) is rated on a scale of 1 to 100.
Each EC member scores the technical proposals in two steps. First, the level of responsiveness of the proposals to each of the criteria or subcriteria is esti- mated on a percentage scale. Second, each percentage rating is multiplied by the maximum number of points assigned to the relevant criterion (or subcrite- rion) in the RFP to obtain the score (percentage rating × maximum number of points = score). For example, the criterion “specific experience of the consultant rel- evant to the assignment” may have been allocated a maximum of 10 points in the RFP. A proposal with a good level of responsiveness to this criterion is given a 90 percent rating and therefore receives a score of 9 points.
To make the scoring easier and more transparent, the rating scale of the level of responsiveness is usually divided into a number of discrete grades. It is a good practice to give scores based on the following grades: poor, satisfactory, good, and very good. Before open- ing the technical proposals, the EC should agree on the definition of a grade for each criterion (or subcri- terion). That is, the committee should establish what will be considered poor, satisfactory, good, and very good. Because each of the criteria (or subcriteria) refers to a different aspect of the proposal, the definition of grades will differ from one criterion to the next.
Scoring technical proposals by this method offers the following advantages:
•
It provides the EC members with a shared defini- tion of the grades, making the evaluation easier and comparable (this is particularly helpful for less- experienced evaluators).•
It reduces the risk of scoring inconsistencies and discretion.•
It binds each committee member to justify his or her individual evaluation, based on a common def- inition of grades, discouraging intentionally biased evaluations.•
It adds transparency and fairness to the evaluation process.Defining the grades is a difficult exercise that re- quires a thorough knowledge of the TOR, the main technical issues to be covered by the consultant as- signment, and the qualifications expected from the consultants. It is worth allocating time and effort to this exercise because it may substantially improve the quality of the evaluation. Rating proposals without
using agreed-on, predefined grades of responsiveness leaves the definition of the grades up to each evalua- tor, making the scoring more likely to be subjective and more difficult to compare.
(The following paragraphs illustrate how to select the rating grades and their definitions. Figure 17.1 illustrates a sample evaluation for one of the five main criteria specified in the RFP.)
17.2 Specific Experience of
Consultants Relevant to the
Assignment (for the FTP only)
17.2.1 Rating Scale
The Consultant Guidelines allow a maximum of 10 points to be allocated to the specific experience of the firm (see table 12.1). Because consultants have been short-listed based on their experience, they are all capable of undertaking the assignment; nevertheless, some specific aspects of the qualifications may make a consultant more suitable than others. Under this cri- terion, Borrowers identify and evaluate those specific aspects.
The grades indicated in table 17.1 are recom- mended as percentage ratings related to the evalua- tion of this criterion. Because consultants have already been short-listed based on their experience, their level of responsiveness to this criterion should not nor- mally be rated less than satisfactory (that is, not under 40 percent).
17.2.2 Aspects to Consider
for the Evaluation
The EC should consider the following aspects in eval- uating the specific experience of the consultants:
•
Experience in Similar Projects. Evidence of havingsuccessfully carried out similar assignments.
•
Experience in Similar Areas and Conditions. The con-sultants have worked in regions or countries with physical, cultural, social, and institutional character- istics comparable to those of the country in which the assignment is to be carried out.
•
Size, Organization, and Management. The consultantshave the capacity (for example, staff, organization, and managerial skills) to carry out the assignment. For some assignments, how long the consultants have been established is to be considered.
•
Specialization. For some assignments, it may be im-portant to evaluate the consultants’ specialized skills and access to particular technologies related to the assignment.
•
Experience in Capacity Building. The consultants’experience in capacity building and transfer of knowl- edge of previous clients’ personnel (if relevant).
•
Quality Management (QM). The availability of awell-established QM system may be taken into ac- count for large and complex assignments.
17.2.3 Defining the Grades
Because subcriteria are usually not provided for this criterion (see para. 12.3), the specific experience will be evaluated as a whole, using the grades set out in table 17.1. An example of the definition of these
grades (based on the aspects listed in para. 17.2.2) is given below (definitions may differ from case to case, depending on the characteristics of the assignment):
•
Satisfactory. The consultants have experience inthe field of assignments similar to the one being
91
EVALUATION PRACTICES OF TECHNICAL PROPOSALS
Defining the Grades 1 7 . 2 . 3