External validity

Yin (2009) defines the external validity as the domain to which the findings can be generalized. Yin (2009) points out that this is an important problem for case studies. The external validity of this study is quite high, because the results are not so much influenced by executing the research in different projects or organizations. The current situation of decision-making can be very different

for other locations or organizations. The change in decision quality due to the use of 3Di will be very different if participants have different reference situations.

This study is at least valid for the situation that is studied:

- Decision-making processes in Amsterdam with involvement of both Waternet and the city

council

- Projects were water management is not seen as the main goal

- Projects with only decision-makers of the municipality, so for example no inhabitants - In the current situation no models or models as Infoworks and WOLK are used

- With use of the 3Di system with the version of June 2014

- A decision-making process with clear steps and a start and end point - An urban storm water project

The sample of workshop participants is not random, however it represents a normal proportion for decision-making processes for urban projects in Amsterdam, because both people from the district council as Waternet are included. The co-organizer of both workshops invited the participants and based the invitations on the proportion in a decision-making system. Only those who were available at the time of the workshops and were interested enough in the subject came to the workshops. Although most invited people attended the workshops, no people from the department of traffic and infrastructure attended. They study is still representative for decision-making processes in Amsterdam, because in a normal decision-making process, these people are also not attending often.

The study is expected to be representative for similar processes in cities in the Netherlands with a similar large (physical) distance between design departments as in Amsterdam. Advantages in communication due the workshops between different organizations or departments are also valid for these cities. However, the study can be less generalized for smaller cities, cities were water and road departments are integrated and municipalities without cities. The scores for consciousness and commitment can deviate because of differences in the decision-making process and communication. The results for the elements information, logic and alternatives can differ if other models as used in Amsterdam (WOLK and Infoworks) are used in the reference situation. If the elements information, logic and alternatives have a higher (or lower) quality in the reference situation, the comparison with 3Di will work out differently. Because of the same reason, most results are not valid for other water simulation models. The amount of information, and the possibilities of other models would be different. However, some other models can provide similar maps with an overview of the flooding problems. These models possibly also affect the decision quality elements consciousness and logic. Besides that, the use of a newer version of the 3Di system might lead to different results, because of changes in the functionalities or calculations.

The research is also valid if other steps of the decision-making process are included. This is because the decision quality of the total decision-making process is simulated by some of the decision- making steps in the workshop. Therefore, the results are less valid for decision-making processes that follow the rounds model and are thus not following steps clearly (Hommes, 2008). Nevertheless, the comparison of alternatives and choosing the final decision are also part of such decision-making processes, so the research results are still valid.

The study is not valid for water projects not concerning storm water. For example for flooding in rural areas, different results can be obtained for the level of detail and the consequences of alternatives, because larger and different alternatives will be used. In addition, the score for coherence between alternatives will be different.

The significance of the results of the interviews is quite low because not so much interviews are done. Especially conclusions about the general decision-making process in storm water management project are hard to see as a firm proof. There are many differences between decision-making processes in different storm water management projects, so this cannot be checked with only a few interviews. The reliability of interviews about the decision-making process in Amsterdam is increased by asking interviewees from different organizations the same questions. In this way, someone’s answer was checked in another interview.

During a congress in Bonn it was suggested that model use can lead to an overload of information. Attendants listened to a presentation about 3Di first, after which a discussion was held. Members of different international municipalities stated that too much information was not useful in a model because that makes it more difficult to understand. This result was not seen in this research, but might be the case if more complicated models are used. Too much information can then lead to less clarity, because it is hard to find the right information. It is therefore not sure if this research is valid for the use of other interactive, detailed and fast water simulation model than 3Di. This could not be checked because there is not much known about such models besides the 3Di system.

Results are consistent with studies about other models, other kind of water project in other countries. Frick & Hegg (2011) found that users will have an increased confidence about situation analysis and decision-making using the D-PHASE platform, a meteorological model. This can be compared with the improved logic in the final decision and the understanding of consequences of alternatives due to using the 3Di model. The information availability, structure and interpretation was also improved by the D-PHASE model of Frick & Hegg (2011), what is similar to the improvement by the 3Di system. The consciousness of the perspective was increased by the use of participatory modeling according to Voinov & Bousquet (2010). According to Leskens et al. (2013), the 3Di system is also increasing the understanding of effects of measures, what is similar to the results of this study. The decision support system AQUATOOL was useful to identify the problem, screening alternatives and for an operational implementation process (Andreu et al., 1996). Arciniegas et al. (2013) found that a high information level of the model results in a lower understanding by the users. This conclusion was not supported by this study, the high amount of information was not seen as a problem of using 3Di. Hwang & Lin (1999) support the conclusion that more information decreases the decision quality, they state that even a small increase of information diversity or dimension can decrease the decision quality.