Improvement phase

The Improvement phase is composed of the set of tasks where the results of the benchmarking process are produced and communicated to the benchmarking partners and the improvement of the different tools is performed in several improvement cycles. In these improvement cycles, improvement is planned, carried out and monitored. These tasks and their interdependencies, shown in figure 4.5, are the following:

I1. Benchmarking report writing.

I2. Communication of findings.

I3. Improvement planning.

I4. Tool improvement.

I5. Monitor.

I1. Benchmarking report writing Actors

Benchmarking teams

Inputs Outputs

Updated benchmarking proposal Benchmarking report Experiment report

In this task, the benchmarking teams must write the report of the bench-marking. Whereas the experiment report provides technical details and results

Figure 4.5: Improvement phase of the benchmarking process.

on the experiment, the benchmarking report is intended to provide an under-standable summary of the process carried out. Thus, the benchmarking report should be written having in mind different audiences: managers, benchmarking teams, developers, etc.

The report should include an explanation of the benchmarking process fol-lowed and all the relevant information from the updated version of the bench-marking proposal; it should also include the results and conclusions of the exper-iments present in the experiment report, highlighting the best practices found during the experimentation and including any best practices found in the com-munity.

The report must also contain the recommendations of the benchmarking team for improving the tools according to the experiment results, the practices found, and the community best practices.

The goal of the benchmarking report is not to provide a ranking of the tools, but to provide the practices and the best practices found in the benchmarking and to give improvement recommendations.

I2. Communication of findings Actors

Benchmarking teams

Inputs Outputs

Benchmarking report Updated benchmarking report Organization support

In this task, the benchmarking teams must communicate the results of the benchmarking study to their organizations, and particularly to all the members involved and identified when planning the benchmarking.

This communication should be in the form of meetings, which should be held for one or more partner organizations. The goals of any benchmarking team in these meetings are twofold:

To obtain feedback from the members concerned of the benchmarking process, results, and improvement recommendations.

To obtain support and commitment from the organization members in order to implement the improvement recommendations in the tool.

Any feedback received during the communications of these findings must be collected, documented, and analysed. Finally, this analysis may result in having to review the work done and to update the benchmarking report.

I3. Improvement planning Actors

Benchmarking teams

Management of the organizations Tool developers

Inputs Outputs

Updated benchmarking report Necessary changes

Monitoring report Improvement planning

Organization support Improvement forecast Measurement mechanisms

The last three tasks of the Improvement phase (Improvement planning, Tool improvement, and Monitor ) form a cycle that must be performed by each or-ganization separately. It must be added that it is from these tasks where each organization benefits from the results obtained in benchmarking.

In this task, the benchmarking team, the managers and the tool developers of each partner organization must identify, from the benchmarking report and the monitoring reports, which are the changes required to obtain an improved tool.

They must also foresee which improvement will be achieved after performing these changes.

Both the organization management and the benchmarking team must pro-vide the organization with mechanisms that ensure the accomplishment of the improvements.

Additionally, the benchmarking team should provide tool developers with mechanisms for measuring tool functionalities. These mechanisms can be ob-tained or adapted from the tools and input data used in the Experiment phase.

Then, they must decide the planning for improving the benchmarked tool and reach a consensus. This planning should be reckoned with and integrated into each organization planning.

The improvement planning should contemplate the time and the organiza-tion resources (people, computers, travel, etc.) devoted to improvement activi-ties.

I4. Tool improvement Actors

Tool developers

Inputs Outputs

Updated benchmarking report Improved tool Necessary changes

Improvement planning Improvement forecast Measurement mechanisms

In this task, the developers of each benchmarked tool should implement the necessary changes in order to achieve the desired results.

But before implementing any changes, the tool developers must assess the actual state of the tool using the measurement mechanisms provided by the benchmarking team in the Improvement planning task. Then, after implement-ing the necessary changes, the developers must measure the tool again and compare the results with both the results obtained before implementing the changes and the improvement foreseen.

I5. Monitor

In each organization, the benchmarking team must provide tool developers with means for monitoring the organization’s tool, such as monitoring tools, benchmark suites, etc.

Tool developers must periodically monitor the tool and write a report with the results of this monitoring.

These monitoring results may show the need for new improvements in the tool and the beginning of a new improvement cycle, which means having to perform again the two previously mentioned tasks: Improvement Planning and Tool improvement.