Evaluate Assessment

The activity in the EABV AP is concerned about the evaluation of the assessment and the information products. This evaluation is done against certain criteria that are aligned with the evaluation criteria identified for our evaluation method 6.5.5. For more on the evaluation of our approach refer to Chapter 6.

5.4.5. Roles and Responsibilities

For us, it is crucial to have clear roles and responsibilities for a process that is integrated into the company’s operations. Recent performance models focus on stakeholders (Kloot and Martin 2000) which underlines the importance of considering people and their relations as asset to deliver performance and consequently business value. These are defined and identified in the

first step of the EABV AP described in Section 5.4.1. Now, that we have outlined all other steps, it is time to examine what various stakeholders have to do in order to perform this assessment in Table 5-8. As can be perceived, we have four different roles for every phase of the EABV AP as well as the related responsibilities. First, we have the project manager, which apart from usual project manager tasks for the assessed project with EA contribution, is responsible that the actual assessment is carried out and therefore must monitor and track it. He takes on the role as the assessment librarian according to the measurement standard ((ISO/IEC 2007)). The scope of this depends where the actual data points, i.e. the input and output for data to the EABV AP, are located over the course of the project. Moreover, a project manager assures adequate project members that are able to collect and provide data. He helps developing the assessment plan and identifying information needs. Additionally, he evaluates the information products and assessment process.

Next, we have project members, responsible for scoping the assessment. In terms of the measurement standard, they are perceived as assessment users. They need to assure they possess the necessary skills to collect and provide data. As with project managers, they assist in creating the assessment plan and identifying information needs. Furthermore, they are responsible for selecting and documenting goals and metrics. Also, they collect and analyse data with predefined and integrated procedures. Finally, they also take part in the evaluation of information products and the assessment process.

EA Managers are responsible for the overall assessment commitment. They secure financial support and allocate adequate EA practitioners to conduct the assessment. EA managers are the equivalent to assessment analysts in terms of the measurement standard. They assure the communication and documentation of information needs and also take part in the development of the assessment plan which they also approve. Moreover, EA managers collect and analyse data and communicate results. As the roles before, they evaluate information products and the assessment process.

EA practitioners assist in identifying the assessment scope and need to be aware of the actual EA contribution. They, like all roles, help to develop the assessment plan. Furthermore, they collect and analyse data after which they are able to develop information products. In addition, they evaluate information products and the assessment process. EA practitioners are, as project members, assessment users according to the measurement standard.

Typically, we find an inter-organizational project setup during an assessment where stakeholders working on the assessed project (project managers and project members, both EA customers) consume EA services and therefore collaborate with stakeholders (EA managers, EA practitioners) providing those services.

EABV AP Role (Project specific) Responsibilities

Establish and Sustain Assessment Commitment Project manager (assessment librarian)

 Overall responsibility that the assessment is conducted

 Define and identify scope of assessment  Allocate relevant project members to collect and

provide data

 Assure that project members are able to collect and provide data in an adequate manner

 Assure communication and initiate contacts between project members and EA practitioners

Project member (assessment user)

 Define and identify scope of assessment

 Assure that skills are appropriate for data collection and providing the data

EA manager (assessment analyst)

 Decision makers in terms of financial support and overall assessment commitment

 Assign EA practitioners to take part in the assessment

EA practitioner (assessment user)

 Define and identify scope for the assessment  Assure that skills are appropriate for data collection

and providing the data

 Be aware of the actual EA contribution

Plan Assessment

Project manager (assessment librarian)

 Make assessment context explicit, including assumptions and constraints

 Identify information needs

 Prioritise and select information needs  Assure communication and documentation of

information needs

 Develop and approve assessment plan  Define information product evaluation criteria  Acquire and deploy supporting technologies

Project member (assessment user)

 Identify information needs

 Select and document goals and metrics  Define data collection, analysis, and reporting

procedures

 Develop assessment plan

 Define information product evaluation criteria

EA manager (assessment analyst)

 Assure communication and documentation of information needs

 Develop and approve assessment plan  Define information product evaluation criteria  Acquire and deploy supporting technologies

EA practitioner (assessment user)

 Select and document goals and metrics  Define data collection, analysis, and reporting

 Develop assessment plan

 Define information product evaluation criteria

Perform Assessment

Project manager (assessment librarian)

 Communicate integrated data collection procedures  Communicate results

Project member (measurement user)

 Integrate data generation and collection  Integrate data analysis and reporting  Collect and analyse data

 Develop information products and communicate results

EA manager (measurement analyst)

 Communicate integrated data collection procedures  Collect and analyse data

 Communicate results

EA practitioner (measurement user)

 Integrate data generation and collection  Integrate data analysis and reporting  Collect and analyse data

 Develop information products and communicate results

Evaluate Assessment

Project manager (measurement librarian)

 Evaluate information product and assessment process

Project member (measurement user)

 Evaluate information product and assessment process

EA manager (measurement analyst)

 Evaluate information product and assessment process

 Identify potential improvements  Communicate potential improvements

EA practitioner (measurement user)

 Evaluate information product and assessment process

 Identify potential improvements

Table 5-8: EABV AP Roles and Responsibilities

Notably, these roles and responsibilities are relevant once the EABV AP is instantiated and working in daily operations. During the design and build phase of this approach, we have different stakeholders and types thereof involved that are responsible for different tasks as previously outlined in Section 2.4.