What is theory? Validity and reliability in project management research

In his article ‘Objectivity, value judgment, and theory choice’ (Kuhn, 1977) Kuhn states there are five characteristics of a good scientific theory, and that a good theory should:

1. Be accurate – consequences deductible from a theory should be in demonstrated agreement with the results of existing experiments and observation

2. Be consistent – not only internally or with itself but also with other currently accepted theories applicable to related aspects of nature

3. Have a broad scope – it should extend far beyond the particular observations, laws, or subtheories it was initially designed to explain

4. Be simple – it should bring order to phenomena that in its absence would be individually isolated and confused as a set

5. ‘Be fruitful of new research findings’ – it should disclose new phenomena or previously unnoted relationships among those already known.

(Tjora, 2012) suggests that new concepts and theories must be falsifiable, and it should at least be possible for other scholars to test and re-examine them. Some scholars argue that it is often quite difficult to falsify and re-examine theories derived from research in which a qualitative approach has been used. Universal theory cannot be made from some observation of a few respondents and based on a limited numbers of cases. However, others argue that case studies and action research are the best way to understand in depth what is going on and do not consider that new concepts and theories based on field studies, case studies, and an action research approach are problematic (Eisenhardt, 1989; Flyvbjerg, 2006; Tjora, 2012).

There are several difficulties relating to research on projects; by definition, they are one of kind – they are always influenced by the time and context in which they are executed and they are always dependent on the behaviour and performance of large numbers of different stakeholder that are directly involved in doing the project or more indirectly involved in the project process. The internal stakeholders can be divided into two groups: workers and managers. Both groups need know their job and deliver if a project is to perform well and succeed with the deliveries. However, most projects have also external stakeholders (e.g. sponsors, project owners, and asset owners) that play a vital role in how the project is carried out. The nature of project as a ‘one-of-a-kind event’ means that much PM theory is based on making sense of limited numbers of observations and that the development of PM theories and concepts will happen in the meeting between scholars and project actors in action. Concept and theory in PM are often built upon observations from cases that have more or less unique contexts and involved different actors. I therefore suggest that it should be accepted that much PM research and theory will be based on some type of qualitative research approach, and that true falsification of PM concepts and theories may be hard or even impossible in some cases, yet that does not mean that such theories have less value or are less valid than theory developed in more quantitative fields.

I suggest that the project management field and tradition have few ‘laws or theorems’ that are accepted as universal. Project management research is often positivistic in its strategy and it pursues generalizations or to establish principles or laws to govern its object (Smyth & Morris, 2007).

However, it is also often qualitative and seeks to understand the world from the perspective of those in it (Merriam, 1995; Neuman, 1997). According to (Lewin, 1946), there are two types of research

objectives in social sciences research: ‘general laws’ and ‘groups of life and diagnostic of specific situation’. Laws deal with possible conditions and possible results, and they are expressed in terms of

‘if so’ propositions and serves as guidance to the achievement of certain objectives under certain conditions (Lewin, 1946). The ‘diagnostic’ has to do with specific character of the studied situation.

Based on Lewin’s ideas, I suggest that project management theories mainly fall into what I class as Level 2 or Level 3 theories, as listed in Table 3.

Table 3 Levels of theory – examples

Level Theory levels Examples

1 Laws and theorems Newton’s laws of motion

First law: ‘An object either remains at rest or continues to move at a constant velocity, unless acted upon by an external force’

P.-S. Laplace – ‘The central limited theorem’

2 Recommended process components

PRINCE2 – ‘the eight management processes’

IPMA’s ‘competence eye’

PMI’s ‘ten management processes’

Project risk management: processes, techniques, and insights (C.

Chapman & Ward, 1996)

‘Practical project management – from idea to profit’ (Rolstadås, 2014)

‘Exploiting opportunities in the uncertainty management’ (Krane, et al., 2014)

‘Opportunities in projects – what are they and do we really want them?’ (Johansen et al., 2012a)

‘Megaprojects – Challenges and lessons learned’ (Zidane, Johansen, Ekambaram, 2012)

‘Uncertainty management – Myths and realities’ (Johansen et al., 2012b)

‘The practical uncertainty management 9 step model’ (Johansen et al., 2013)

I consider textbooks and standards such as the PMBOK (PMBoK, 2000), APM BoK (Dixon, 2000), PRINCE2 (Paul, 2003), IPMA competence Baseline (Association, 2006), ISO 21500(Norma; Zandhuis &

Stellingwerf, 2013), that have been developed over decades as based on Level 2 theories. It may eventually be possible to falsify some part of this work and thereby consider it as based on Level 1 theories. I place from scholars’ and practitioners’ papers, models, and theoretical contributions in an early stage of development on Level 3. In addition, I consider much of the theoretical contribution of this thesis as Level 3. Hopefully, some of the concepts and theoretical contributions in this thesis also have high external validity and reliability, such that the concepts may change or influence recommended theoretical practice at Level 2.

Validity and reliability in project management research

The majority of the theoretical foundations of the project management field are theory and concepts developed from some type of observations of actions from projects when they are executed or some type of summarizing of data from projects after they had finished. The qualitative research approach

is often preferred in PM research (Smyth & Morris, 2007). The approach is ideal when operative variables cannot be identified ahead of time, and when the purpose of the study is to understand how the different actors in the project perceive their roles or tasks. The qualitative research approach is often the preferred strategy if the researcher is more interested in trying to build hypotheses rather than testing them (Merriam, 1995).

How the validity (true measured) and reliability (dependable measured) of the research conducted on project is judged is linked to the research strategy (i.e. quantitative versus qualitative approach).

(Neuman, 1997) divides validity into internal and external validity, and argues that internal validity has to do with few or no errors in the internal design of the research project. High internal validity means that there are few errors in the design of the study and low internal validity means that internal errors are likely and may have affected the study. External validity is related to the ability to generalize findings from a specific setting, small samples, and a small or wider group of people. High external validity means that the results can be generalized to many groups and many projects, whereas low external validity means that the results apply only to a specific situation. According to (Neuman, 1997), relatability means dependability or consistency, and expresses how easy or hard it will be for other researchers to repeat a study and its conditions, and to what extent the findings will be repeated (Eisenhardt, 1989; Merriam, 1995; Neuman, 1997; Yin, 2003). The more often the findings of a study can be replicated, the more stable and the phenomenon will be considered (Merriam, 1995).

Project management is a part of the social science tradition and qualitative methods are often the preferred research method (Smyth & Morris, 2007), which means that high reliability is normal also problematic in project management research. Perfected reliability in the sense of whether another researcher can replicate a study and find the same results is impossible in most project management research because projects are of one kind. (Merriam, 1995) cites Lincoln and Guba (1985), who state that ‘the real question of qualitative research is not whether the result of one study is the same as the results of a second or third study, but whether the results of a study are consistent with the data collected’ (Merriam, 1995). Users of qualitative method in PM research are likely to assume that projects are constructed phenomena that are multidimensional and depend on the actors involved, the context in which they are executed, and researchers’ capability to interpret the data and studied phenomena. Merriam states:

Qualitative researchers are not seeking to establish ‘laws’ in which reliability of observations and measurements are essential. Rather qualitative researchers seek to understand the world from the perspective of those in it. Since there are many perspectives, and many possible interpretations, there is no benchmark by which on can take repeated measures and establish reliability in the traditional sense. (Merriam, 1995)

Several strategies can be chosen to strengthen the internal and external validity and reliability in project management research (Eisenhardt, 1989; Merriam, 1995; Neuman, 1997; Yin, 2003):

Strategies for strengthening internal and external validity:

x Triangulation – use of multiple investigators, multiple sources of data, or multiple methods to confirm the findings

x Member checks – test and check the data collected on the people from whom they were derived; present the tentative interpretations of the data on some of the actors from the study

x Peer examination – test on colleagues the data and plausibility from the study

x Statement of researcher’s experience and background, as well as statements of the starting point of the research, biases in literature, and the study cases or examples.

x Submersion or engagement in the research situation – collect data from enough projects or cases over a long enough period to ensure that in-depth understanding of the phenomenon is obtained.

Strategies for strengthening the reliability of a study:

x Clear conceptualization of the study – have a clear theoretical starting point and clear measures that indicate that only one concept is preferred, if possible

x Multiple levels of measurement – try to measure at the most precise and most specific level as possible

x Use multiple indicators of a variable – two indicators tend to be more stable than one x Use pre-tests or pilot studies – by testing questions or doing a pilot study, ‘bad questions’

and errors in the study can be corrected before conducting the full-scale study.

High validity (internal and external) and reliability should always be the goal when research is conducted. However, in project management research it must accepted that most of the research is conducted as snapshot of a moving object, and that in most cases there are limited possibilities to control the context, the stakeholders, and all of the parameters that influence the study. It is therefore important that it is made clear how the study was conducted, what the limitations of the study are, and how far it is possible to generalize the results of the study.