Subject selection used a combination of self selection and accidental selection. An initial list was drawn up of organizations located in Sydney and known to develop software. Each
organization was phoned and a request was made to speak to a project manager. Sometimes this request resulted in an explanation to an R&D manager or the divisional secretary or someone similarly unconnected with software development. But, generally, people were prepared to listen and tried to accommodate the request. Six of the initial list of eighteen organizations accepted and were subsequently interviewed.
Accidental selection was performed through phoning organizations listed in the “Computer Software and Packages” section of the Sydney Yellow Pages. No record was kept of how many organizations were approached. However, after reviewing the retained section of the Yellow Pages, I estimate that approximately 400 organizations were approached. Thirty two interviews
Page 110
were conducted with people from twenty eight organizations. The acceptance rate isapproximately 7%.
4.7 External validity
External validity is the extent to which the findings may be generalised to other contexts (Leedy and Ormrod, 2001). In turn, this is related to the sample and its selection. The method of
selecting the sample has been discussed (Section 4.6), so this section will examine the sample characteristics to establish how well the sample represents the general population of software development organizations.
As can be readily seen from the data presented in the analysis (Section 5.3), the sample is not representative of all software development organizations. This is evident in the distribution of organization sizes in the sample which shows a bias toward small organizations and
multinational organizations (Table 24 in Section 5.3). Similarly, the weak external validity is evident when comparing the distribution of organization process capability level in the sample with the distribution of the same characteristic over a wider population gained during trials of the SPICE standard (Table 25 in section 5.3). Thus extreme care must be taken with
externalising the findings of this research outside the research sample.
The sample size of 32 project managers belonging to 28 different organizations is small for quantitative research. While it may be large enough to perform some simple statistical tests such as Pearson’s correlation or a Chi-square test, it is not large enough for any multivariate analysis. Even within the sample, some of the tests did not have sufficient occurrences in each category to get a result.
Despite this weak external validity, the results presented in Chapter 5 are reasonably
unambiguous throughout the sample. This indicates that similar results would have been found had the sample been larger and more representative of software development projects. The conclusions, discussed in Chapter 6, are likely to be valid for any software development project in Australia.
4.8 Construct validity
Construct validity is the extent to which the research instrument measures a characteristic that cannot be directly observed (Leedy and Ormrod, 2001). It is also expressed in an ISO standard on measurement as the “fitness for purpose” of a measure which is judged by how well it measures what it purports to measure (ISO 15939, 2002).
Page 111
The number of interview questions related to each construct was reviewed to ensure thatsufficient data would be gathered and that the questions were not overlapping or unnecessarily repetitious. Keeping the questions in a database made this task relatively easy because the list of questions could be reported, grouped by construct, by category or in interview sequence.
Questions could also be reported listing which construct they supported to check which were the important questions – the ones that supported multiple constructs.
The constructs needed to investigate these questions are described in the following section. Some of the constructs are nominal and some ordinal.
4.8.1.1 Organizational Distance
This is an ordinal construct made up of three dimensions: cultural distance, structural distance and administrative distance.
Cultural distance borrows from Hofstede and the work of others (Hofstede, 1983a; Ronen and Shenkar, 1985; Hofstede, 1991; Shenkar, 2001). This research uses the clusters of similar national cultures proposed by Ronen and Shenkar (1985) to classify cultural distance. Structural distance was judged by a combination of geographical distance and time zone difference. Geographically separated countries that did not have a large time zone separation were considered closer than those that did have a large time zone difference. So Japan was considered much closer to Sydney than India.
Administrative distance was measured by the number of organizational layers between the project manager and the project team. This was assessed by whether the project manager had direct access to the project team or was constrained to deal through the outsourced
organization’s project manager, or through someone like a marketing manager who was removed from the outsourced organization’s project manager. This is intended to be a coarse measure of how much direct control a project manager has over the task. Techniques of direct supervision are different from those of indirect supervision. This variable is intended to indicate distance between the project manager and the person or team carrying out the project task. 4.8.1.2 Project management activities
This ordinal construct is intended to discover the range of activities used by project managers to monitor, control and coordinate a task. Lacking a predefined set of activities, the questions must be designed to elicit and accept a range of responses that may be analysed later and possibly group the responses into similar categories. From there, the mechanism being employed by the project manager can be deduced. For example, if a project manager said they dealt with a task overrun by rescheduling the project then the activity is rescheduling but the mechanism is the schedule.
Page 112
Asking a project manager directly how they monitored the project or controlled the project team may evoke a range of responses that are not directly comparable. Different project managers may have different ideas of what “monitor” is or what “control” is. Project coordination, for example, is achieved indirectly and one project manager might consider a particular action related to coordination while another might consider it related to project control. It is also possible that a project manager might consider that some actions are related to risk management and have nothing to do with monitoring, control or coordination. To overcome these different perspectives, project managers were presented with a situation and asked what they would do about it.Acknowledging that the different project management mechanisms are inter-related and that a question directed at, say, monitoring is likely to evoke information relating to monitoring as well as control and coordination, interview questions were aimed at the different project
management mechanisms as follows. Questions 16, 17, 18, 30 and 31 (Appendix A - Structured interview questions.) elicit information on project monitoring. Questions 2, 8, 9, 10, 11 and 25 - 29 relate to project control. Questions 19, 20, 21, 22, 33 - 36 relate to project coordination. 4.8.1.3 Project management capability
Project management capability is an ordinal construct modelled on a SPICE process capability. There may be variations of how well the project monitoring and management is performed. Such variations should conform to a form of capability scale. The concept of process capability is taken from software process assessment models such as CMMI (SEI, 2000) and SPICE (ISO 15504, 1998). In such models a process is defined in terms of its purpose and outcomes. The outcomes provide evidence that the process purpose is being achieved. In turn, evidence that outcomes have been achieved must be objective and requires either the production of a deliverable work product or a significant change of state (of a deliverable or work product). Emphasis is on objective evidence. A change of state could be that a document changes state from "draft" to "reviewed" or from "reviewed" to "approved”. The set of outcomes is considered as basic to the process. The objective evidence attests that the outcomes have been achieved, which demonstrates that the process purpose has been achieved. Additionally, there are a set of outcomes related to how well the process is managed and the achievement of these outcomes determines the organization’s process capability or maturity.
Where possible, the fully tested SPICE indicators of process capability were used but, where no such indicators were available or gave no result, alternative indicators of capability were developed.
Page 113
4.8.1.4 Organizational characteristics
Organizational characteristics such as size, type of software developed and quality certification were solicited so that some additional analysis could be performed. These attributes were either ordinal (size, process capability) or nominal (existence of process improvement programme). 4.8.1.5 Project characteristics
The attributes of a “typical” project were characterised by an ordinal measure of size and a nominal classification of system/industry type
4.9 Internal Validity
Internal validity is the extent to which the data allow the researcher to draw accurate
conclusions about cause and effect (Leedy and Ormrod, 2001) and other relationships within the data. In this section, the validity of the ordinal constructs of organizational distance and project management capability is discussed.
Threats to internal validity can arise through, inter alia, reactivity, experimenter expectancy (Leedy and Ormrod, 2001 p104). Although the research participants were told that the research subject was how project managers managed their projects and, in particular, how they managed distributed and outsourced projects, there was no discussion of the expected results. This was simply because, while interviews were being conducted, no conclusion had been reached about what were the likely answers to the research questions.
However, a threat to internal validity arises because participants were chosen through accidental sampling. Some organizations declined to participate in the research and those that did
participate could be disposed toward particular project management mechanisms.
4.10 Conclusion
Research described here is best performed from a pragmatic knowledge claim that allows and supports mixed method research. This allows data to be gathered for later quantitative analysis for similarities or trends across all cases and for reviewing the qualitative data for unanticipated insights.
Existing empirical studies of project management mechanisms have been conducted using case studies, surveys, interviews and controlled experiments. This established a range of appropriate research methods to guide decisions for this research. A strategy was chosen that acknowledged the researcher’s limited experience and the probability that both knowledge and perception of the subject was likely to change during the research. This resulted in choosing structured
Page 114
interview as a method of gathering both quantitative and qualitative data and providingsufficient samples to provide some, albeit limited, statistical validity.
A questionnaire was developed to guide the interview. Its composition was guided by the information needed for the constructs of interest to this research. The questionnaire was modelled on methods of assessing organizational process capability that have been used in the software development industry since 1993. The research instrument (the questionnaire) was reviewed during a doctoral assessment and after the initial few interviews. Modifications were made to the research instrument that clarified some interview questions without altering the validity or utility of data already gathered.
The external validity of the research is limited because the research sample is comparatively small at 32 instances, and because the characteristics of the research sample differs from the characteristics of the general population of software developers. Construct validity is claimed through a review of the information gathered for each construct but is not proven through any external test. Such tests of validity are beyond the scope of this current research project. The research data were gathered by interviewing software development project managers, guided by the research instrument. The interviews were audio recorded, transcribed and checked by the interviewed project manager. The results of the analysis form the focus of Chapter 5.
Page 115
5 Analysis
The previous chapter considered the research methodology (survey) and research method (structured interview) best suited to investigate the research questions. This chapter will examine the data thus collected. First, the sample characteristics are examined. Then, the data will be examined to establish how project managers generally monitor, control and coordinate their projects. Then, the measure of organizational distance, and the data contributing to the measure, will be presented. This allows the distribution of the research sample on the scale of organizational distance to be discussed. Next, the effect of increasing organizational distance on the selection and use of monitoring, controlling and coordinating mechanisms will be examined. Each mechanism will be examined in turn to establish whether different mechanisms are
associated with different organizational distances. Then, an analysis of which mechanisms are essential for project management of software development projects will be presented.