Australian Software Development: What Software Project Management
Practices Lead to Success?
J. M. Verner
National ICT Australia
Australian Technology Park
Alexandria
Sydney, Australia
june.verner@nicta.com.au
N. Cerpa
Departamento de Ingeniería de Sistemas,
Universidad de Talca,
Camino Los Niches Km. 1,
Curicó, Chile
ncerpa@utalca.cl
Abstract
We surveyed a number of Australian software practitioners in order to understand what software development practices were used in their recent software projects. We were particularly interested to discover what project management practices are common in Australian software projects. The relationship between practices and software project outcomes enables us to investigate why some projects succeed and others fail. We found that nearly 20% of projects had no lifecycle methodology and 10% of our respondents did not understand what was meant by a software development lifecycle methodology. Many recognized software practices are not being applied consistently in the projects investigated. Fifty percent of projects began with unclear requirements. Risk assessment is not normally a part of the development process and the organizations are not learning from their mistakes as post mortem reviews are much more likely to be held for successful projects than they are for unsuccessful projects.
1. Introduction
In order to gain an understanding of Australian software development practices and to identify possible research opportunities we used a previously developed questionnaire [20] to collect data from Australian software developers about software projects on which they had recently worked. The developers we contacted were asked to describe a recent IT project, either successful or
random sample but rather a convenience sample of practitioners known to us.
The questionnaire was organized into seven sections; however owing to space limitations we only consider three sections related to project management here:
• Project management, • Requirements, and
• Cost/effort estimation and scheduling
The project management perspective is important because most software engineering research has emphasized “technical matters above behavioral matters” [10]. Moreover, “there has been a general lack of quantitative survey-based research regarding early non-technical aspects of software development”.
The motivation for this work is to explore project management practices in order to provide a preliminary set of project success factors for Australian software project managers. This will give general guidance for both project managers and business managers to help ensure that the software development projects they are responsible for succeed.
We discuss, in Section 2, details of the questionnaire and characteristics of the project data, in Section 3, the results of our survey are described, and in Section 4, we identify and discuss areas for further research.
2. Questionnaire Responses
We received completed questionnaires from 42 Sydney software developers each reporting on a different software development or maintenance/ enhancement project (i.e., 42 projects). The
majority of our respondents were developers involved with software for use within their own organizations (financial institutions, banks, pharmaceutical companies, insurance companies, etc.).
3. Results and Analysis
Our sample size is quite small but large enough to gain some useful insights into both Australian software development practices and to identify promising areas for further research. Of the projects our respondents chose to describe, 78% were successful, 78% were development rather than maintenance/enhancement projects, 66% of the development projects were successful and 88% of the maintenance/enhancement projects were successful.
The percentages of full time employees on the projects was 1-4 = 29; 5-9 = 37; 10-19 = 16; 20-29 = 2; 30-39 = 5.5; 40-99 = 13; and 100-180 = 2.5 (range 1-103, median 7).
We asked our respondents what development methodology was used; 19.5% of the projects had no prescribed development methodology, 31% used a waterfall or a modified waterfall methodology, 2% used a prototyping approach, 2% the spiral model, and 10% an in-house proprietary methodology. Twenty six percent of respondents did not answer this question and the rest (9.5%) apparently did not understand what was meant by a software development methodology. In response to the question “Was a defined software development lifecycle methodology (your own or another) used? Explain what this methodology was”, comments such as “GUI based”, “procedural way”, “simple method to acquire requirements”, “users + stakeholders”, “user workshops”, “tick sheets listing all tasks required to be done” etc., were made. No respondents reported on a project that used incremental delivery.
The percentage of “yes” responses to the survey questions is shown in Table 1. Table 2 shows those variables with a significant relationship with project success (<0.05 in a chi square test, with correlational analysis providing the direction of the relationship). Five aspects of project management are shown: the project manager (prefixed with “M” in column one), requirements analysis (“R”), cost/schedule estimation (“C”), risk assessment (“A”), and post-mortems (“P”). In the rest of this paper when we comment on the relationships between variables, these relationships are significant in a chi square
test at < 0.05.
3.1. Project Management
Since we assume that project management and the project manager (PM) are pivotal to success or failure of a software development project, we begin with an analysis of PM participation in the process. We would expect that a software development project would have a project manager; no Australian projects were without an assigned project manager. Other research [20] has noted that some software development projects have proceeded without an assigned project manager with an expectation that project management will be spread between the managers of the involved functional groups within the organization. The project management tasks are supposedly shared equally. Reports on this type of arrangement are not encouraging [20].
In 22% of the Australian projects the PM was changed at least once. This volatility, practitioners reported, was very disruptive. The largest Australian project surveyed was different from results reported elsewhere [20] where the largest projects were more likely to have the project manager changed; our project had 100 internal staff and 3 contractors, the PM who had 15 years’ experience was not changed and everyone considered the project a success. Changing the PM was not significantly correlated with project success.
The PMs’ years of software development experience ranged from under 6 months to 20 years (median of 5). This variable was not correlated with project success. We found that 58% of project managers had a software development background, 32% a business background, and the rest had various backgrounds (engineering etc.). Neither a PM with a software development background (M14) nor one experienced in the application area (M2) was significantly correlated with project success. This result is in agreement with observations that a broad background is more useful than expertise in any particular technical area. “Successful project managers are generalists, not technical specialists”; while a certain level of technical competence is helpful, managerial and interpersonal skills are more important [11]. Other results support this perspective since the ability to communicate (M5) and relate well with staff (M6) were both positively correlated with project success.
Table 1: Percentage “Yes” responses to questions ID Question Success1 % Yes Failure2 % Yes Total3 % Yes
M0 Did the project have a project manager (PM)? 100 100 100
M1 The PM was above average? 71 33 63
M2 Was PM experienced in the application area? 72 56 68
M3 Did PM understand the customer’s problems? 83 44 74
M4 Did PM have a clear vision of the project? 81 78 80
M5 Did PM communicate very well with the staff? 84 33 73
M6 Did PM relate well with the staff? 74 33 65
M7 Did the PM delegate authority? 97 78 93
M8 Did PM pitch in and help when necessary? 84 67 80
M9 Did PM control the project? 88 67 83
M10 Did PM treat the staff equally? 81 56 76
M11 Did PM appreciate staff for working long hours? 84 11 68
M12 Were staff rewarded for working long hours? 53 22 46
M13 Was the project manager changed during the project? 19 33 22 M14 Did the project manager have a software development background? 56 67 59 R1 Were requirements gathered using a specific method? 59 56 59 R2 Were requirements complete and accurate at project start? 53 56 54 R3 If not complete at start were requirements completed later? 87 0 68
R4 Overall, were the requirements good? 94 56 85
R5 Did the project have a well defined scope? 84 78 83
R6 Did the scope increase during the project? 50 67 46
R7 Did the users make adequate time available for requirements gathering? 69 56 66
R8 Was there a central repository for requirements? 87 56 79
R9 Did requirements result in well defined deliverables? 87 44 77 R10 Did the size of the project have an impact on requirements? 47 44 46 C! Was the project manager involved in making initial cot/schedule estimates 59 22 50 C2 Was the delivery decision made with appropriate requirements information? 56 33 51
C3 Did the project have a schedule? 97 100 98
C4 If yes, were the estimates of effort and schedule good? 53 22 46
C5 Were the developers involved in making estimates? 72 56 68
C6 Did the project have adequate staff to meet the schedule? 69 44 63 C7 Were staff added late to meet an aggressive schedule? 25 67 34 A1 Were potential risks identified at the start of the project? 75 67 73
A2 Were risks incorporated into the project plan? 66 22 56
A3 Were the risks managed throughout the project? 59 44 56
P1 Was a post-mortem review held? 52 11 43
P2 There was a post mortem and results of a post-mortem were made available to other groups?
48 17 41
1 This column represents the percentage of “yes” answers to questions for successful projects. 2 This column represents the percentage of “yes” answers to questions for projects that were failures. 3 This column represents the percentage of “yes” answers to the questions for all projects.
Table 2: Relationships of questions to project success
(ns = not significant)
ID Question Direction of Success
Relationship
Association with project success M0 Did the project have a project manager (PM)?
M1 The PM was above average? + 0.004
M2 Was PM experienced in the application area? ns
M3 Did PM understand the customer’s problems? + 0.032
M4 Did PM have a clear vision of the project? ns
M5 Did PM communicate well with the staff? + 0.007
M6 Did PM relate well to the development staff? + 0.033
M7 Did the PM delegate authority? ns
M8 Did PM pitch in and help when necessary? ns
M9 Did PM control the project? ns
M10 Did PM treat the staff equally? ns
M11 Did PM appreciate staff for working long hours? + 0.000
M12 Were staff rewarded for working long hours? ns
M13 Was the project manager changed during the project?
ns M14 Did the project manager have a software
development background?
ns R1 Were requirements gathered using a specific
method?
ns R2 Were requirements complete and accurate at
project start?
ns
R3 Were requirements completed during the project? + 0.004
R4 Overall, were the requirements good? + 0.015
R5 Did the project have a well defined scope? ns
R6 Did the scope increase during the project? ns
R7 Did users make adequate time available for requirements gathering?
ns
R8 Was there a central repository for requirements? ns
R9 Did requirements result in well defined deliverables?
+ 0.018
R10 Did the size of the project have an impact on requirements?
ns C1 Was project manager involved in making initial
cost/effort estimates?
ns C2 Was delivery decision made with appropriate
requirements information?
ns
C3 Did the project have a schedule? ns
C4 If yes, were the estimates of effort and schedule good?
ns C5 At some stage, were the developers involved in
making the estimates?
ns C6 Did the project have adequate staff to meet the
schedule?
ns C7 Were staff added late to meet an aggressive
schedule?
- 0.029
A1 Were potential risks identified at the start of the project?
ns
A2 Were risks incorporated into the project plan? ns
A3 Were the risks managed throughout the project? ns
P1 Was a post-mortem review held? NA
P2 If there was a post-mortem review, were results made available to other groups?
A PM rated above average (M1) is positively correlated with project success; this result is not surprising since “poor management can increase software costs more rapidly than any other factor”
[1]. An above average PM (M1) is involved with good schedule estimates (C4) made with appropriate requirements information (C2).
Our data supports the view that management support of the development team is essential to motivate the team to work effectively toward organizational goals [7] as appreciation for working long hours (M11) was positively correlated with project success. Interestingly, rewards for staff who worked long hours (M12), was not correlated with project success. These results are quite different from those in an analysis of US data reported in [20]. Here appreciation for working long hours, and rewards for working long hours were both significantly correlated with project success. Are Australian developers unconcerned with financial rewards but rather expect that their PMs acknowledged if they work long hours? Do Australian software developers actually expect to have a life outside of work and are financial rewards less important than they are to their US counterparts? An Australian Software project manager commented that “I consider on-going acknowledgement of the hard work done by my team is an important part of good project management.”
Using logistic regression the best PM predictor of project success was M11 (Did the PM appreciate the staff for working long hours), which predicted 89% successful projects, 83% failures and 84% correctly overall. We required in our logistic regressions that the explanatory variable have at least a 5% level of significance.
3.2. Requirements
Good project management necessitates that requirements are complete and consistent [18]. We had expected that gathering requirements using a specific methodology (R1) would provide a better chance of developing good requirements and hence a higher likelihood of success. However, R1 is not significantly correlated with project success. Anyhow, in 71% of our projects respondents did not know what requirements methodology was used. We were not surprised by this as most respondents did not have interaction with the project’s systems analysts. For the small number of projects where respondents did know about the requirements gathering, one project used prototyping; 3 projects used JAD sessions with
interviews and focus groups as the main requirements gathering method. No projects used UML to document requirements. The unsuccessful projects included one that used JAD.
Because nearly half the projects began with incomplete requirements (R2) it is not surprising that the scope was changed for many projects (R6); aȤ2 test of R2 with R6 was significant. The scope
was also more likely to change for larger projects. These results suggest that organizations are having some problems with their software requirements. Given that control over the requirements function is necessary to move from the lowest CMMI level, it is obvious that many of the organizations in our sample are still at the lowest level [5]. These results agree with [17], whose respondents thought that their companies did not do enough requirements engineering. The number of projects that began with poor requirements suggests that the organizations should be developing more of their projects with software development methodologies designed to deal with incomplete requirements, (such as prototyping or incremental development); however, this is not happening.
Our results, shown in Tables 1 and 2, demonstrate that requirements continue to be a problem for IS development [16, 19]. Not surprisingly, and consistent with observations made by Glass [8] we found that good requirements (R4) that had been completed at some stage during the project (R3) resulting in well-defined software deliverables (R9), were positively correlated with project success. Although Boehm [2] includes a “continuing stream of requirements changes” in his top ten risk items, we did not find that changing the scope during the project (R6) was (negatively) correlated with project success. Instead, we found that if requirements were initially incomplete, completing the requirements during the project (R3) was positively correlated with project success. Over 70% of projects used a central repository for their requirements and this suggests that most of the Australian organizations are starting to take the management of their project requirements seriously. Although research [8] suggests that project size hampers requirements gathering, and leads to unclear, incomplete, and, potentially, unstable requirements we did not find a significant correlation between large projects and project success. However, given that there were so few projects employing over 50 developers we do not have enough data to properly test this. Of the three largest projects two succeeded and one failed.
Using logistic regression, the best predictor of project success was R4 (the requirements were
failures, and 78% correctly overall. 3.3. Cost/Schedule Estimation
Common wisdom suggests that good estimates of effort and schedule (C4) affect project success [7]. As early as 1975, Brooks stated that more projects have gone awry for lack of calendar time than from all other causes combined [3]. Optimistic estimation is still one of the two most common causes for runaway projects [9] with cost and schedule failures exceeding any other kinds of software failures in practice [8]. Boehm [2] includes unrealistic schedules and budgets in his top 10 risk items. However, having a schedule (C3) was not correlated with project success; not having a schedule alleviates the problem of meeting a schedule.
We initially assumed that the PM would be involved in deciding the delivery date as he/she is likely to have better knowledge than anyone else of the technology, the people, and the development practices used for the project. However, this assumption was correct in just over 50% of cases; initial estimates were made by the PM alone in 26% of cases, it was a committee or a group decision (involving the PM) in 28% of projects, the customer in 5% of the projects and higher-level management, or marketing, without PM involvement in 41% of projects. Responses to this question were not correlated with project success. PMs were able to negotiate the schedule in only one third of the projects where they were not included in the initial delivery-date (or budget) decisions. These results agree with Glass [9] and Yourdon [21]. When we look at projects that had poor estimates of effort and schedule they were fairly evenly split between those that had PM input and those that did not. Given the lack of PM participation in the estimation process, it is not surprising that only about half the projects had a delivery date made with appropriate requirements information (C2). Our result agrees with Glass [9] that most software estimates are performed at the beginning of the lifecycle before the requirements phase and thus before the problem is understood.
Fifty four percent of projects were underestimated, 41% were accurately estimated, and 5% projects were overestimated. Overall, respondents thought that 15% of estimates were either poor or very poor (50% of these projects were successful), 39% were average (75% successful) and 47% were either good or very good (89% successful). Many estimates (63%) initially thought to be of average quality were underestimates while the projects that were
overestimated were initially thought to be good or very good estimates. Developer involvement in making the estimates (C5) was not correlated with project success. Developers were not more likely to be asked to contribute to project estimates when the requirements were poor so at least they would not be blamed for inaccurate estimates caused by incomplete requirements. This result is quite different from that reported in Verner and Evanco [20] where the developers were only asked to help with project estimates when the requirements were incomplete; in this earlier research developer involvement was significantly correlated with project failure. Larger projects were more likely to have staff added late. Finally, the only variable in the cost and schedule set of data significantly (negatively) correlated with project success, in agreement with [3] was C7 (adding staff added late to meet an aggressive schedule).
Using logistic regression, none of the estimation variables were good predictors of project success.
3.4. Risk Assessment and Post-Mortem Reviews
Unfortunately, most developers and project managers perceive risk management processes and activities as extra work and expense [14], with risk management the least practiced discipline amongst the different project management areas [15]. The size of the project made no difference to whether or not risks were incorporated into their project plans. Managing risks throughout the project was not correlated with project success [7]. The quality of software project management is characterized by active risk management [13, 18]; this observation is not supported by a correlation between responses to question A1, A2 and A3 and the PM being above average (M1), though other researchers have reported such a correlation [20].
Post mortem reviews are important for process improvement [9] but they are seldom done. As a result, we tend to repeat the same mistakes on project after project [4, 6, 10, 12]. Few project post mortems were held. For organizations that do conduct them, reviews are not consistently done [20]. Holding post-mortem reviews (P1) was significantly correlated with good requirements (R4) as well as all the risk variables, i.e., risks identified at the beginning of the project, risks incorporated into the project plan, and managing risks throughout the process (A1, A2, A3).
4. Discussion
Surveys are of course based on self-reported data which reflects what people say happened, not what they actually did or experienced. Because we surveyed software developers our results are limited to their knowledge, attitudes, and beliefs regarding the projects, the actual projects they chose to report on and PMs with which they were involved. However, as the majority of projects are fairly small (66% employed fewer than 10 people and 82% fewer than 20), we believe that our respondents have a reasonable knowledge of most project events. The overall preponderance of small projects may however, bias our results. The developers we surveyed mainly develop in-house software for their organization’s use. Their organizations have a heavy reliance on software for many business functions. While we cannot assume that our results are typical of all Australian organizations, the data has provided us with some questions that require further research.
We were surprised that nearly 50% of projects started with unclear requirements. Why are PMs prepared to go ahead with projects without either appropriate requirements or a lifecycle methodology able to deal with unclear requirements? It is common wisdom that good requirements lead to software development success so why are PMs apparently so unaware that they are prepared to jeopardize project success in this fashion? Poor requirements are likely to have a negative effect on the estimation process; this then leads to schedule and cost underestimates, and adding staff late.
Business management seems to lack an appreciation of the steps necessary to successfully execute a project. In one sixth of our projects we found that the PM was effectively shut out of the cost estimation and scheduling process. They were not only excluded from initial discussions but were subsequently not permitted to negotiate what had been decided by business managers. Consequently, some PMs are short-circuited in the management of their projects. Senior management needs better education regarding the importance of adequate requirements, systematic effort and schedule estimates and the importance of consulting PMs for projects with fixed budgets. Developer input into the estimates did not appear to make any difference to the quality of the estimates.
There is a mismatch between risk identification and control. While the majority of project managers identified risks at the project start, just over half followed through during development.
Yet risk management sets up projects for success [7].
In most organizations each project is viewed as a standalone entity so that post mortem reviews are not perceived as important. Both business managers and project managers do not appear to understand the specific causes of failed projects and consequently are unlikely improve their performance on subsequent projects. Finding out what went wrong with a project ought to be seen as a good thing not something that must be buried. Given that over 50% of the successful projects had post mortems some organizations like to find out what went right during a project and what methods and procedures should be institutionalized. However, with post mortems on only 11% of the failed project learning from mistakes is limited. If PMs are unable to admit to, or investigate failure then their projects are likely to continue to fail.
The best predictor of successful project outcomes, using logistic regression, M11 (Did the PM appreciate the staff for working long hours), which, as noted earlier, predicted 89% successful projects, 83% failures and 84% correctly overall. This result is quite different from other research that investigated US in-house software development and found that factors such as 1) a PM who has a clear vision of the project, 2) a delivery decision made with appropriate requirements information and 3) project had good requirements, were the best predictors of projects success. However, with our small data set our results suggest that further research is required to determine if Australian software development projects have different success factors from projects conducted in other countries.
Table 1 shows that current practices are reasonable and in some areas rather better than practices reported on projects from the US [20]. The opportunity for greatest improvement is at the start of a project in the requirements and project risk identification and control areas. Here many of the factors are practiced in fewer than 75% of projects. These issues are very important if we wish to increase the quality and success of software project management.
Analysis of our survey suggests further research is required in order to investigate:
• Are these results really representative of Australian software projects?
• Why is some common terminology, such as a software development lifecycle methodology, not understood by a proportion of developers? • Are Australian developers more interested in
• What kinds of pressures lead PMs to start projects with poor requirements, and continue without an appropriate lifecycle methodology that will compensate for this?
• Why are risk mitigation and control practices so frequently ignored?
• Why are so few post mortem reviews conducted on failed projects? Are we more interested in gaining kudos for the things we did well than in finding out what we did badly?
5. References
[1] Boehm B. W., Software Engineering Economics, Prentice Hall, Englewood Cliffs, NJ, 1981.
[2] Boehm B. W., “Software Risk Management Principles and Practice”, IEEE Software, Jan/Feb, 1991, pp. 32-41.
[3] Brooks F. P. Jr., The Mythical Man Month. Essays on Software Engineering, Addison Wesley, USA, 1975.
[4] Brossler P. “Knowledge Management at a Software House: A progress report” Proc Workshop on Learning Organizations, 1999, pp. 77-83.
[5] CMMI http://www.sei.cmu.edu/cmmi/ accessed 5th July 2004.
[6] Collier B, T. DeMarco and P. Fearey, “A Defined Process for Project Postmortem Review”, IEEE Software, July/August, 1996, pp. 65-72.
[7] DeMarco T. and T. Lister, Waltzing With Bears, Dorset House Publishing New York NY, 2003. [8] Glass R “Error-Free Software Remains Extremely
Elusive”, IEEE Software, Jan/Feb, 2003, pp. 103, 104.
[9] Glass R. L., “Frequently Forgotten Fundamental Facts about Software Engineering” IEEE Software, July/August, 2001, pp. 112, 110, 111.
[10] Glass R. L., “Project Retrospectives, and Why They Never Happen”, IEEE Software, September/October, 2002, pp. 112, 111.
[11] Jurison, J., Communications of AIS, Software Project Management, The Manager’s View, Volume 2, 1999, Article 17.
[12] Kerth, N. L., Project Retrospectives: A Handbook for Team Reviews, Dorset House Publishing, New York 2001.
[13] Keuffel, W., “Planning for and mitigating risk” Software Development Vol. 7, No 9, 1999, pp. 81-85.
[14] Kwak, Y. H. and Stoddard J., “Project risk management: Lessons learned from software development environments”, Technovation, 2004. [15] Kwak, Y. H. and Ibbs, C. W. “Calculating project
management’s return on investment” Project Management Journal, Vol 31, No 2, 2000, pp. 38-47 [16] Moynihan T., “How experienced Project Managers Assess Risk”, IEEE Software, May/June, 1997, pp. 35-41.
[17] Neill C. J. and Laplante P. A., Requirements Engineering: State of the Practice” IEEE Software, November/December, 2003, pp. 40-45.
[18] Osmundson J. S., Michael J. B., Machniak, M. J., and Grossman M. A., “Quality management metrics for software development” Information and Management, 2003.
[19] Schenk, K.D and Vitalari, N. P., et al. “Differences Between Novice and Expert Systems Analysts: What Do We Know and What Do We Do?”, Journal of Management Information Systems, Vol. 15, Issue 1, Summer 1998, pp.9-51.
[20] Verner J. M. and W. M. Evanco “In–house Software Development: What Software Project Management Practices Lead to Success?” IEEE Software, Jan/Feb, 2005, pp. 86-93.
[21]Yourdon E., “A tale of two futures” IEEE Software January-February 1998, pp23-29.
