Plan Improve

Evaluate

Portfolio & project management Ad hoc  Formalise  Institutionalise

“Practice and Play”

Evaluation planning Evaluation workshop

assessment provide the input for the later activities of the Evaluate stage and during the Plan stage of the methodology.

6.2.1.2.1 Introduction and Context

The session commences with a very brief presentation with the sole purpose of clarify potentially unclear aspects within the questionnaire. No information on the ICMM and improvement methodology is offered to avoid any potential influencing and biasing of responses. The participants should respond to the questions as spontaneously as possible (avoiding over-rationalisation and hidden agenda such as considering how one may answer to benefit ones selves). Information provided includes certain definitions of words that have specific meanings within the questionnaire.

The most important part of this introductory presentation is providing the context for answering the questionnaire. The context specifies how participants are to relate the questions to their organisation and what specifically is being assessed – the organisation as a whole, a particular business unit, etc. This would have been established in the earlier meeting with management for instance. The reason why this context is so important is because it provides a common basis for respondents to complete the questionnaire. If one participant considers the Sales Department, while another, the organisation as a whole; their responses become incomparable. The problem is that the analyst (who processes these responses) probably would not even know. It is, therefore, essential that this context be emphasised in the introductory presentation.

6.2.1.2.2 Completion of questionnaire

Once the presentation has been completed and any questions from the participants answered, the participants are introduced to the questionnaire. The first section, the respondents‟ general information, is completed without (much) guidance.

The next section, however, requires emphasis to ensure that the participants provide answers that are as accurate as possible. This section, the roles description (see Section 6.1.4), provides the analyst with the necessary understanding of the each of the respondents‟ role within the organisation to better interpret their responses and appropriately calculate the results. Participants are, therefore, carefully guided through this part of the questionnaire.

The remained of the questionnaire, the innovation status description and the 42 Capability Requirement questions, are fairly self explanatory and, therefore, completed by the participant alone. However, a facilitator is required throughout the session to answer any queries that the participants may have.

6.2.1.2.3 Discussions between participants

Apart from the innovation capability maturity results that are presented in a later session, the biggest short- term benefit from such an exercise is the discussion that comes of it. This session and the questionnaire are intended to stimulate discussion, as participants have their responses and relative perspectives of the

strengths and weaknesses and the possible reasons therefore. Many participants from the case studies (Chapter 7) stated that they found the discussions that ensued as a result of the evaluation process (both in the session and thereafter) interesting and relevant to the challenges that they face as an organisation. It is, thus, crucial that the facilitator (and the analyst) take as many notes as possible while the participants are discussing these aspects. This is particularly important if the analyst has had limited previous exposure to the organisation.

6.2.1.3

Processing of responses

An important aspect of an Innovation Capability Maturity evaluation (and any evaluation based on a questionnaire) is the interpretation of the questionnaires and the translation of answers into value-adding and descriptive results that improve the respondents‟ understanding of the situation being evaluated.

Once the evaluation workshop has been completed, the analyst captures the responses and calculates the results, i.e. determines the innovation capability of the organisation (or business unit). However, very little can be extracted from the single maturity rating – the results must be presented at various levels of aggregation and from multiple perspectives. This is necessary to identify the innovation capability strengths and weaknesses of the organisation. In terms of results, the following levels of aggregation may be obtained:

 Overall Innovation Capability Maturity – the highest level of detail, a single rating between 1 and 5 signifying the overall innovation capability maturity of the assessed organisation (referred to as overall, normalised average innovation capability maturity level in the case studies).

 Innovation Capability Areas – the 3 high-level areas of innovation capability represent the second level of detail. A single rating between 1 and 5 is provided for each, signifying the organisation‟s ability to fulfil the Innovation Process, Knowledge & Competency and Organisational Support requirements.

 Innovation Capability- and Organisational Construct Items – the 11 and 6 respective construct items for innovation capability and the organisation represent the third level of detail. Again, each construct item receives a single rating.

 Innovation Capability Requirements – the lowest level of detail, the 42 capability requirements each receive a maturity rating between 1 and 5 depicting the level at which the organisation fulfils the requirements.

In addition to these levels of aggregation, the results may take on various perspectives that include:

 Participants-based – the results based on the raw responses of participants, i.e., averaged and consolidated without any biasing or normalisation.

based on their innovation role profiles captured in the questionnaire.

 Other (management, years in organisation, etc.) – various other views on the results at the request of participants and relevant stakeholders. For instance, in Case Study 2, the view of management was compared to that of the other members of the organisation. The differences between these 2 views highlighted potential communication gaps between the 2 parties (see Section 7.1 for more details).

Figure 35 demonstrates the combinations of results that may be obtained when combining the levels of aggregation and perspectives. Depending on how many “Other” perspectives there are, the total number of different perspectives that could be obtained and analysed is large. This reiterates the importance of determining the specific requirements of the organisation during the initial planning activity.

Note that the perspectives in Figure 35 (participant- and role-based and other) are depicted as mutually exclusive views on the data. This is not always the case. For instance, consider the view of all participants and the view of management – the later is essentially a subset of the former. The figure merely demonstrates that different views may be taken on the same data leading to different depictions of organisational innovation capability.

The ability to take these “Other” views depends on the data captured in the general information section of the questionnaire. It is, therefore, recommended that this be discussed with the instigators of the evaluation before the workshop is conducted, i.e., during the evaluation planning (as discussed in Section 6.2.1.1). It may in certain circumstance be possible to obtain the necessary data after the fact, although this is not ideal as it will mean re-establishing contact with all the participants unnecessarily.