Project performance and maturity regarding uncertainty

The research findings relating to the studies discussed in Section 3.5 revealed that almost all organizations involved in the research had increased the maturity level of their organizational project uncertainty management. In order to see whether the long-term efforts had had any impact, we needed to be able to measure whether there had been any changes or developments over the time period to which our research results related. We had close contact with six major organizations in Norway through the PUS project, which provided us with a vast amount of data. To help us narrow down the scope, we used the following four elements in our analyses of the organizations’ maturity level:

x The use of the term ‘uncertainty’ should include both opportunities and threats on equal terms, and both aspects should be exploited and visible in the projects and in the management of the projects from the organization’s view.

x There should be a clearly structured process; in other words, the organizations should have established what to do, when it should be done, and who would be responsible.

x There should be a clear recommendation of tools and techniques that projects should apply in their daily management.

x There should be a clear structure for support, training, and further development of the uncertainty management process in the projects and in the line organization.

We anticipated that all four aspects would be covered if they were considered as indicative of mature organization in uncertainty management. In addition, we anticipated that an organization with high maturity on uncertainty would have a clear process that included opportunities and threats, and that both aspects would be exploited and visible in the management of the projects from the organization.

We also wanted to elaborate more on how the organizations involved in the PUS project developed and how they matured within project uncertainty management during the time they were involved in the PUS project. Kerzner (2009) has described a maturity model in connection with project management (see Figure 13). This model can be used to look at the levels at which an organization learns and develops.

Figure 13 The five levels of maturity (Kerzner, 2009)

Based on Kerzner’s maturity model, we tried to make a subjective evaluation of the maturity levels in the organizations involved in the PUS project. We followed these organizations closely for four years, and felt confident when trying to illustrate their maturity level in the same manner. Table 8 shows the maturity level of the involved organizations regarding managing uncertainty in projects at the beginning of the PUS project in 2006 (denoted by a small ‘x’), and the maturity level of the organizations regarding managing uncertainty in projects at the end of the project in 2010 (denoted by a large ‘X’).

Table 8 Subjective evaluation of maturity level of uncertainty management in 2006 and in 2010 (A. Johansen, Langlo 2013)

Company Level of maturity*

1 2 3 4 5

Statoil (private) xX

Telenor (private) x X

Norwegian Directorate of Public Construction and Property Management (Statsbygg)

x X

Norwegian Armed Forces (Forsvaret) xX

Norwegian Public Roads Administration (Statens vegvesen)

x X

Norwegian government agency for railway services (Jernbaneverket)

x X

Note: *The five levels of maturity as derived from Kerzner’s maturity model (Kerzner, 2009); x

=2006 ; X =2010

From Table 8, it is clear that Statoil was considered a matured organization in terms of dealing with uncertainty management from the beginning of the PUS project. They had a clear understanding of the importance of uncertainty management in their projects. Statoil had made systematic efforts to develop their workforce (humans), models, and techniques, where developed. They were (and still are) at a level where they performed ‘living uncertainty management’; they were constantly working to improve their knowledge, skills, attitudes, processes, and tools in order to enhance their uncertainty management process.

When it comes to the Norwegian Armed Forces, the PUS project did not have any noticeable impact.

This may have been due to the recurring restructuring processes that would have taken away or reduced the needed attention and energy to work with managing uncertainty in projects.

The Norwegian Directorate of Public Construction and Property Management improved its ability to manage uncertainty in its projects through its participation in the PUS project. The organization established new roles (such as uncertainty coordinators), conducted courses for its employees to improve their knowledge and competence, developed its model for dealing with uncertainty management, and started to apply systematic techniques to improve uncertainty management. In addition, the organization’s participation in the PUS project and the good results were recognized in a high profile manner when a prize for innovation was awarded to its representatives in 2011.

Norwegian Public Road Administration made developments in managing uncertainty. Conducting and arranging courses for its employees, developing and applying models and techniques (e.g. tools for analysis) were prominent focal points that the organization started to adopt in order to manage uncertainty effectively. This description also points to the role of ‘living uncertainty management’ in the organizational setting.

The improvement that Telenor and Norwegian rail administration experienced can also be considered with respect to the ‘living uncertainty management’. The focus on humans, models, and techniques, and the interplay between these three elements may have contributed to create a positive culture that promotes effective management of uncertainty. The wishes and intentions to create a positive culture for uncertainty management was mentioned by the representatives of organizations that participated in the PUS project (e.g. by Telenor).

In the longitudinal study carried out in the PUS project, the respondents were asked to explain what they based their execution of uncertainty management on.

Figure 14 The basis for execution of uncertainty management – responses from a survey in 2010. (A. Johansen, Langlo 2013)

Figure 14 is based on a survey of all the partners in the PUS project. The Figure shows that projects utilized gut feeling in combination with tools and routines. This is a good indication of maturity with reference to the fact that both tools and gut feeling are used as equal early warning signs in projects.

(Klakegg, Williams, Walker, Andersen, & Magnussen, 2010). Two of the investigated organizations scored slightly differently compared to the others, which was natural as all of the projects had different maturity levels. The fact that some of the organizations had inconclusive results regarding which of the three alternatives they used is also an indication of lower maturity. Potential reasons for this were lack of training, no common tools for uncertainty management, wanting the involvement of the project owner, and lack of procedures for uncertainty management, just to mention a few. Figure 14 shows a snapshot in time, and that there were clear differences between the evaluated organizations, but it does not provide any clues as to how this situation evolved from 2006 to 2011, which is illustrated in Figure 15.

Figure 15 The basis for execution of uncertainty management – developments from 2006 to 2011 (A. Johansen, Langlo 2013)

Figure 15 shows how the score for the same question had changed between 2006 and 2011. This is was the average score for all organizations. From the Figure it is apparent that the single use of gut feeling had decreased and that there was an increase in the combined use of tools and gut feeling.

This is an indication that the average project uncertainty management maturity had increased from 2006 to 2011. Figure 14 does not show the development for each organization, but from our detailed results it was clear that all organizations had improved; some had improved more than the others, but all had improved. In order to investigate the maturity further, we asked the respondents whether it was acceptable in their organization to express their concerns regarding uncertainty. Figure 16 shows how their attitudes had changed between the period 2006–2007 and the period 2010–2011.

Figure 16 Changes in attitude towards expressing uncertainty (A. Johansen, Langlo 2013)

Figure 16 illustrates that all organizations largely accepted and endorsed project team members’

opportunities to express their concerns. The change between the two periods as significant, as the score for (2) had reduced from 37% to 18%, while the score for (1) had increased from 47% to 76%.

This indicates that uncertainty management maturity had increased in all organizations. However, there was still room for further improvement. In this case too, the results are the combined results for all the organizations, and the detailed results showed that some organizations had more improvement than others. Still, there was some degree of improvement in all organizations.

Finding 2: Increased ability to meet cost targets

A further main finding related to the studies discussed in Section 3.5 was documented through an evaluation done by the Norwegian Ministry of Finance in 2012. The results of the evaluation indicate that to a large extent recent Norwegian public projects have been able to meet their cost and time targets. Furthermore, since the estimates had some variance, the Ministry of Finance was interested in measuring how much variation they could find in the final cost compared to the control estimate.

Figure 17 was produced in 2012 by the Ministry and shows the cost deviation for 35 of the then most recent investment projects.

Figure 17 Cost deviation relative to P85 estimates (presented by O.J. Klakegg at the Concept Symposium in 2012)

It is interesting to observe that the distribution of the 35 projects that Klakegg analysed in the Concepts study in 2010 presented almost identical shape as to a P85 distribution. The standard in Norwegian public projects is to use P85 estimates as the total budget, and Figure 17 shows approximately 85% of the projects were spending less than their budget. We found the same situation when we compared the results with the P50 estimates: approximately 50% of the projects were spending less than their budget. This indicates that public projects in Norway had succeeded in their estimation and project control, and that their uncertainty management processes had been successful. We also found that most of the projects were returning 5–30% or more of their budget.

This could indicate that they had overestimated, but it could also indicate that they were looking for opportunities and actively exploiting them, thus returning money to the government. Public projects

do not have any natural incentives to reduce cost, as the project does not benefit from this itself. The results presented by the Ministry of Finance are therefore a strong indication that there has been a shift in the mindset of public projects, and that uncertainty management is placed higher on the agenda. It is also an indication that the public agencies are taking a clearer lead in the improvement efforts and that they are succeeding in completing projects within budget and within time.

Finding 3: Increased active involvement of the project owner in uncertainty management

The research results of the studies discussed in Section 3.5 documented that project owners had become more actively involved in managing the uncertainty in their projects. This aspect is important to ensure that the project can utilize the opportunities discovered in course of project execution. We found that also the project owners involved in the PUS project had improved their processes, procedures, and tools for project uncertainty management (Ekambaram A, 2013b; A. Ekambaram, Johansen, Aalstad, & Hansen; A. Johansen, Halvorsen, Haddadic, & Langlo, 2014; Langlo, et al., 2007;

N. O. Olsson, Johansen, Langlo, & Torp, 2008). Some of the project owners had actively developed and offered internal training in uncertainty management, both formal (together with academic institutions) and informal (in-house). We also registered that there was more communication between the project and the project owner than before, and that the project owner was taking more part in following up the project.

However, it is important that the project owner is ‘hands on’ and not ‘hands in’. There is a balance to be maintained, and if the project owner is too involved in the project, he or she will undermine the position of the project manager. Nevertheless, the project owner has to be involved and have close relations with the project manager and the project if opportunities are to be exploited and when measures on major threats are to be developed and implemented. Both cases normally mean that the scope has to be changed, that funds have to be redistributed, or that the cost will increase. All these changes require acceptance from the project owner, and project owner involvement will ensure project success. It is therefore with great enthusiasm we registered that in recent years project owners had taken a more active part in the projects and were ‘playing’ their role better than before.

Finding 4: Potential negative impacts

The research results of the studies discussed in Section 3.5 also unveiled some unintended and potential negative impacts of the improvement efforts, which may have resulted in more expensive projects and less innovation.

As mentioned previously, it is easier to meet a budget if the cost is overestimated. In the first years of the quality assurance regime, we detected a certain increase in the budget, and almost all of the external reviews resulted in increases in budget, both in the base estimate, but more frequently in the uncertainty allowances. This could have been a result of previous underestimations of the cost of uncertainty, but it is also likely that all parties benefited from increasing their budgets: the project, because it would be easier to ensure success; the politicians, since they would not have to answer for cost overruns to the opposition; and the external reviewers, since they would not like to be blamed for cost overruns, and so forth. It became apparent though, that this development could have two major disadvantages: (1) the tax payers would have to pay more for the services provided by the public as the projects became more expensive, and (2) the large allocation of funds to meet potential

threats from all the projects combined could actually reduce the total number of projects to be sanctioned in a given time period.

There are at least three possible explanations for why projects in Norway, at least on paper, have fewer cost overruns compared to what seems to be the average elsewhere in the Western world.

First, they may have been some ‘hidden reserves’ in the estimates that project manager team did not know about – the base estimate should be without any contingence according to the textbooks.

However, it is almost impossible to check whether some of the participants in the process had added some contingence into the different estimates as a part of the preparation of the cost statement that served as input to the uncertainty analysis workshop.

Second, the ‘good’ project managers chose the contractor that had bid that was 20–25% lower than what the project manager anticipated would be the end result of the contract. If the project anticipates that the highest a contract could end on is 120, the project manage must secure a bid on 100 so that the contingency will still remain intact and can be used on uncertainties that will occur in the execution phase of the project.

Third, in reality the budget may have been higher than a P50 estimate when the project started the execution phase. If contracts increases by 15% to 20% on average, as companies in Norway experience, is it not possible to manage the uncertainty with an 8–10% contingency allowance on project management level. Hence, if they do not choose a bid that is lower than the expected end value or if they have hidden reserves in the estimate they will need to raise the total budget to a higher level than P50. That means that instead of a P50 estimate the project will in fact have the P60 or P65 estimate as their budget, but this is not expressed or communicated to the project owners.

Despite the positive indications, we considered that these potential negative impacts still have to be watched closely the coming years, and it should always be a part of the project owner’s responsibility to make sure that the funds are appropriate for the project objectives.

Finding 5: Overestimation and underestimation of potential influence

The study done by Klakegg et al. shows that there is still room for improvement (Klakegg et al., 2010).

Figure 18 illustrates the score of important factors at the time of cost estimation (to the left in blue) and the real score at project completion (to the right in orange). Klakegg et al.’s study suggests that the project’s management team overestimates the potential influence on the project from some factors, whereas the influence from other factors is underestimated.

Figure 18 Important factors influencing the project – anticipated versus real (presented by O.J.

Klakegg at the Concept Symposium in 2012)

It is known that over the years organizations and the market situation will always become the top priority at the time of quality assurance before project sanction. It was therefore interesting to find that they still came out on top, but at a much lower impact than anticipated (Figure 18). It seems that the actual most influential factors are overrated, and that the factors coming out as less important at the time of quality assurance are underrated. In other words, there is clearly room for improvement.

This indicates that the uncertainty analysis systematically delivers results that take too much headroom for the most frequent factors, and too little headroom for the apparently less frequent factors. This finding is supported by the line of thinking related to the concept of ‘black swans’. These findings will certainly foster more research on uncertainty management.

The results of the study conducted by Klakegg et al. (2010), evaluating 23 capital projects in Norway, indicate that the approximately 85% of the projects are spending less than their budget. When compared with the P50 estimates, the authors found the same situation: approximately 50% of the projects were spending less than their budget. The projects’ cost control was so good that most of them returned 5–30% (or more) of their budget. Klakegg et al.’s study indicates that public projects in Norway are succeeding in their estimations and becoming better at cost control, and that their uncertainty management processes are successful. However, this could also indicate that they are projects that overestimated in the early phase, or looked for opportunities and actively exploited them, thus returning money to the government. A public project does not have any natural incentives to reduce cost, as it will not benefit from cost or time reduction itself. The results presented by the Ministry of Finance are therefore a strong indication that there has been a shift in the mindset of public projects, and that uncertainty management is placed higher on the agenda. It is also an indication that the public agencies are taking a clearer lead in the improvement efforts and that they are succeeding in completing projects within budget and within time.

Finding 6: The ‘blind spot’ of uncertainty management

The research done by the PUS project has revealed something that could be called the ‘blind spot’ of uncertainty management. In short, the blind spot it the lack of ability to unveil and utilize the opportunities in a project. Figure 19 illustrates the development in number of opportunities and threats during project planning and execution phases. In the left-hand part of the Figure, the

development regarding opportunities and threats is symmetrical, but with an overall, though uneven, dropping trend towards delivery. In the right-hand part of the Figure, the set of lost (i.e. unexploited)

development regarding opportunities and threats is symmetrical, but with an overall, though uneven, dropping trend towards delivery. In the right-hand part of the Figure, the set of lost (i.e. unexploited)