Software Project Management Tools
Draft 5.0 June 28, 2013
Keywords: Software project management, software sizing, software cost estimating, software schedule planning, software quality estimating, software risk analysis, software progress tracking, software litigation, software management tools.
Abstract
Software projects have been notoriously difficult to manage well. Cost overruns, schedule delays, and outright cancellations of software projects have been common occurrences for more than 50 years. Poor quality when software is delivered remains a chronic issue. Although software project management tools are not a panacea, when used by capable managers they do simplify the complexity of software project management and raise the odds of successful completions of large software projects.
Software project management tools have many divisions and subcategories. However they can be broadly classified into two major areas of focus: 1) Tools for planning and estimating software projects before they begin; 2) Tools for tracking and reporting software project status and costs while projects are underway. Beneath this division are 12 topical areas and scores of individual project management activities.
Note that the distinction between software projects and systems engineering projects is ambiguous. Note also that modern Software as a Service (SaaS) and cloud computing projects require interfaces among numerous software systems.
Capers Jones, Vice President and CTO, Namcook Analytics LLC
Web: http://www.Namcook.com
Blog: http://NamcookAnalytics.com Email: [email protected]
Software Project Management Tools
The Tasks of Software Project Managers
This article discusses the tasks of software project management and the available tools that support those tasks. From the frequent failure of large software projects and from analysis of the depositions and testimony of litigation for software failures, it is apparent that software project managers are often not well trained in managerial tasks, and often fail to utilize appropriate tools to support critical tasks such as estimating and progress tracking.
Problems, failures, and litigation are directly proportional to the overall size of software applications measured using function point metrics. Table 1 shows the approximate distribution of software project results circa 2013:
Table 1: Normal Software Results based on Application Size
Note: Costs are based on $10,000 per month
Size in Schedule Total Productivity Cost in U.S. Odds of Odds of Function in calendar Staffing in Function Dollars Project Outsource
Points months Points per Failure Litigation
Staff Month 1 0.02 1 50.00 $200 0.10% 0.00% 10 0.40 1 25.00 $4,000 1.00% 0.01% 100 3.50 2 14.29 $70,000 2.50% 0.25% 1,000 15.00 6 11.11 $900,000 11.00% 1.20% 10,000 35.00 50 5.71 $17,500,000 31.00% 7.50% 100,000 60.00 575 2.90 $345,000,000 47.50% 23.00%
As can be seen from the distressing percentages of failures and litigation for software projects larger than 10,000 function points, the successful development of large applications remains troublesome even in 2013.
Before discussing the tools of software project management, it is appropriate to consider the many tasks that software project mangers are likely to confront in carrying out their jobs. The two major divisions of software project management include planning and estimating before projects start, and tracking status, progress, and expenses while software projects are underway. Beneath this broad division there are a dozen major functional areas and scores of individual tasks, as shown below in table 2:
Table 2: The Tasks of Software Project Management 1) Software Project Sizing
a. Size in function points of new development b. Size in function points of legacy components
c. Size in alternate metrics (story points, use-case points, etc.) d. Size in lines of code (logical and physical)
e. Size in data base contents f. Numbers of screens g. Numbers of reports h. Numbers of interfaces
i. Number of associated applications j. Numbers of files
k. Prediction of requirements creep l. Prediction of requirements churn
m. Prediction of deferred requirements slippage to future releases n. Sprint contents for Agile development
2) Software Project Schedule Planning
a. Project schedules b. Activity schedules c. Task schedules
d. Overlap and concurrency of scheduled task e. Time splitting among key technical workers f. Sprint schedules for Agile projects
g. Work-breakdown analysis h. Critical paths
i. Earned value j. Team assignments k. Individual assignments l. Requirements creep m. Requirements churn
n. Requirements slip to future releases o. Requirements cancellation
3) Software Project Cost Estimating
a. Estimating method selection b. Estimating tool selection
c. Historical data analysis from similar projects d. Project cost estimating
e. Phase cost estimating f. Activity cost estimating g. Task cost estimating
h. Reusable material estimating
i. Commercial off the shelf (COTS) acquisition estimating j. Open-source acquisition estimating
k. Project earned value calculation l. Creeping requirements cost prediction m. Churned requirements cost prediction n. Slipped requirements cost prediction o. Cancelled requirements cost accumulation p. Complexity estimating
r. Special estimates for FDA and FAA certification
s. Special estimates for COTS integration and ERP integration t. Special estimates for data migration from legacy applications
4) Software Project Risk Estimating
a. Sarbanes-Oxley risks for financial applications by large companies b. Governance criteria for key financial applications
c. Participation in software process assessments
d. Risks associated with low levels of capability maturity e. Risks of serious problems with COTS applications f. Risks of serious problems with open-source applications g. Risks of serious problems with cloud-computing components h. Risks of serious problems with SaaS components
i. Risks of serious problems with legacy components j. Risks of serious problems with hardware components k. Risks of serious problems with reused components l. Risks of theft of intellectual property
m. Risks of schedule delays n. Risks of cost overruns o. Risks of project cancellation p. Risks of poor quality
q. Risks of poor customer satisfaction
r. Risks of loss of stakeholder or sponsor support s. Risks of poor team morale
t. Risks of hardware changes during development
v. Risks of litigation for breach of contract
w. Risks of litigation for intellectual property issues and patent issues x. Risks of loss of key personnel
y. Risks of strikes by union personnel z. Risks of loss of key clients
aa. Risks of unexpected competitive applications in same field bb. Risk of business failure due to recession
cc. Risk of layoffs due to recession
dd. Risks of natural disasters (flu epidemic; weather, etc.) ee. Potential security problems – prevention
ff. Potential security problems – active defenses gg. Potential security problems - recovery
5) Software Project Quality Estimating
a. Defect potentials for application under development b. Defect potentials for legacy components
c. Defect potentials for COTS packages d. Defect potentials for reusable components
e. Defect potentials for SaaS links and Cloud applications f. Defect potentials for open-source applications
g. Defect potentials for hardware platforms h. Defect prevention methods
i. Defect removal activities
j. Defect removal efficiency (DRE) for inspections k. Defect removal efficiency (DRE) for static analysis
l. Defect removal efficiency (DRE) from independent verification and validation (IV&V)
m. Code coverage from static analysis
n. Defect removal efficiency (DRE) for audits
o. Defect removal efficiency (DRE) for test stages (up to 20 forms of testing) p. Defect tracking tools and protocols
q. Bad fix injections r. Error-prone modules s. Bad test cases
t. Cyclomatic and essential complexity u. Number of defect removal activities v. Number of test stages
w. Number of test scripts
x. Number of test cases per stage
y. Test coverage of applications and features z. Human factors and usability defect removal aa. Nationalization and translation defect removal bb. User guide and HELP text defect removal
6) Software Project Outsource Contract Analysis
a. Requirements scope in contract b. Requirements changes in contract
c. COTS package responsibilities in contract d. Quality terms in contract
e. Defect removal efficiency (DRE) in contract f. Security criteria in contract
g. Intellectual property and non-disclosure in contract h. Incentives for early completion
i. Penalties for late completion j. Penalties for poor quality k. Required tracking and reporting
l. Service-level agreements (SLA) in contract
m. Usage of Information Technology Infrastructure Library (ITIL)
7) Software Project Commercial Off-the-Shelf (COTS) Acquisition
a. COTS acquisition analysis b. Open-source acquisition analysis c. SaaS acquisition analysis
d. Cloud computing acquisition analysis e. Due diligence prior to major acquisitions f. Warranty provisions by vendors
g. Installation schedules
h. Installation assistance from vendor or consultants i. Installation defects
j. Learning curves
k. Loss of performance during learning
8) Software Personnel Management
a. Specialist selection b. Hiring of new employees
c. Employee confidentiality agreements d. Training and education of employees
f. Appraisals of existing employees g. Promotions h. Demotions i. Terminations (voluntary) j. Terminations (involuntary) k. Awards
l. Patents filed by employees
m. Books and articles written by employees
9) Software Project Measurement, Tracking, and Control
a. Financial tracking b. Earned value tracking c. Balanced score card tracking d. Goal-question metric tracking e. Milestone tracking
f. Issue and “red flag” tracking g. Risk tracking
h. Quality and defect tracking i. Security vulnerability tracking j. Change request tracking k. Requirements creep tracking l. Requirements churn tracking
m. Requirements slippage or cancellation tracking n. Complexity measurement
o. Benchmark productivity data p. Benchmark quality data
q. Coordination with other managers in-house r. Coordination with sponsors and stakeholders
s. Coordination with related hardware or system managers t. Coordination with Quality Assurance managers
u. Coordination with supply chain managers v. Coordination with SaaS and cloud organizations w. Coordination with user associations (if any)
10) Software Technology Acquisition
a. Evaluation of proposed tools and methods b. Licensing patents and intellectual property c. Project planning tools
d. Project estimating tools e. Project tracking tools f. Defect tracking tools g. Complexity analysis tools h. Requirements methodologies i. Design methodologies j. Development methodologies
k. Training in new methodologies, if any
l. Maintenance and enhancement methodologies m. Renovation of legacy applications prior to use n. Customer-support methodologies
o. Quality control methodologies p. Methodology management tools q. Development workbenches
r. Maintenance workbenches s. Renovation workbenches t. Reverse engineering tools u. Reengineering tools v. Programming languages
11) Software Mergers and Acquisitions
a. Due diligence for purchaser
b. Analysis of patents and intellectual property in company to be acquired c. Analysis of major applications owned by company to be acquired d. Analysis of tools in company to be acquired
e. Analysis of portfolio in company to be acquired f. Analysis of litigation against company to be acquired g. Analysis of litigation by company to be acquired h. Technology conversion after merger
i. Portfolio consolidation after mergers j. Organization planning after merger k. Reorganization after merger
l. Layoffs and downsizing after merger
12) Software Litigation Support
a. Breach of contract litigation
b. Intellectual property and patent litigation
c. Employee confidentiality and non-competition litigation d. Preparation of materials on plaintiff side
e. Preparation of materials on defendant side f. Discovery documents if required
g. Depositions if required h. Testimony if required
As of 2013 automated tools are available for many but not all of the tasks that software project managers are likely to perform. For common tasks such as schedule planning and cost estimating there are dozens of available tools. For less common tasks such as litigation support or due diligence prior to a merger or acquisition, few tools are available.
Software project management tools are evolving as new kinds of applications are developed. They are also evolving to support new kinds of software development methods such as Agile development, the Rational Unified Process (RUP), and the Team Software Process (TSP), among others.
Project Management Knowledge Acquisition
Academic training in the area of software project management is not as sophisticated as academic training in basic software engineering or even in other forms of engineering management. For example there is a shortage of academic training even for important topics such as quality estimation, risk analysis, and milestone tracking. For more unusual topics, such as litigation support or commercial off the shelf software (COTS) acquisition academic training hardly exists at all.
Because academic training tends to lag the state of the art, newer topics such as cloud computing, Software as a Service (SaaS), and even Agile development usually have no academic courses for perhaps five years after the technologies start use in industry. This is a normal situation because experience must be accumulated before it can be taught.
The lack of academic training for software project managers is compensated for in some degree by training and “bodies of knowledge” or BOK provided by a number of professional associations. Among the large associations that provide useful information to software project managers can be found:
Academy of Management (AOM) www.aomonline.org
American Management Association (AMA) www.amanet.org
American Society for Quality (ASQ) www.asq.org
Certified Enterprise Risk Management Academy (CERM) www.cerm.org
IEEE Computer Society www.computer.org
Information Technology Metrics and Productivity Institute (ITMPI) www.itmpi.org
International Function Point Users Group (IFPUG) www.ifpug.org
International Project Management Association (IPMA) www.ipma.org
International Organization for Standards (ISO) www.iso.org
Project Management Association of Japan www.pmaj.or.jp
International Software Benchmark Standards Group (ISBSG) www.isbsg.org
Project Management Institute (PMI) www.pmi.org
Software Engineering Institute (SEI) www.sei.org
Society of Project Management (SPM) www.spm-japan.jp
There are also scores of national and local organizations that also provide information and seminars on software project management topics. Examples include the Australian Institute of Project Management & Australian Software Metrics Association & Software Quality Assurance (ASMA-SQA), the Brazilian Association for Project Management, the Netherlands Software Metrics Association (NESMA), and dozens of local chapters of the Software Process Improvement Network (SPIN) in the United States, Europe, South America, and Asia. A search of the web will turn up at least 50 national software project management associations.
Several web sites contain useful links to the project management literature. Among the commercial links can be found the previously mentioned Information Technology Metrics and Productivity Institute ITMPI portal whose URL is www.itmpi.org.
Among academic links one of the most complete is that of Dave W. Farthing of the University of Glamorgan in the United Kingdom (www.comp.glam.ac.uk). This interesting portal has links to dozens of project management sites and the publishers of scores of project management books. The author of this paper, Capers Jones, VP and CTO of Namcook Analytics LLC, also has extensive publications on software project management topics including several popular books such as Patterns of Software Systems Failure and Success, Applied Software Measurement, Estimating Software Costs, Software Engineering Best Practices, and The Economics of Software Quality.
The author and Namcook have also published several hundred monographs and journal articles, including an article in Scientific American Magazine on sizing software applications using functional metrics. This is one of very few software articles published in Scientific American.
From discussions with project managers in large corporations, in-house training of managers in companies such as IBM, Microsoft, AT&T, Google, and CISCO is actually the most effective source of information for software project managers. Topics such as quality estimation, measurement, and risk analysis are often better covered via in-house training than by universities.
However this form of training is only available for project managers in rather large corporations. Small companies must depend upon commercial training or upon professional associations, plus a few universities that do have curricula in software management topics.
In recent years new forms of management training have started to take advantage of the web. Almost every day there are webinars and podcasts on project management topics. In addition many companies such as IBM, Rational (an IBM subsidiary), Microsoft, MicroFocus, Computer Aid Inc., the Information Technology Metrics and Productivity Institute (ITMPI), Namcook Analytics LLC, and scores of others offer on-line training in dozens of managerial topics.
In recent years new forms of software such as Software as a Service (SaaS), cloud computing, and the integration of open-source applications and commercial applications into private application has raised the complexity levels of project management. Some projects circa 2013 involve coordination among perhaps a dozen companies and more than 25 components whose development may be taking place in different countries.
Training in specific project management tools is also available from the vendors themselves. For example fairly sophisticated training in software cost estimating is available from the vendors of commercial estimating tools such as KnowledgePlan, Price-S, SLIM, SEER, and Software Risk Master (SRM). Computer Aid Inc. and its subsidiary the Information Systems Metrics and Productivity Institute (ITMPI) also provide training in a number of software management topics. Many of their courses are certified by the Project Management Institute (PMI).
Overall, training of software project managers is a topic that needs to be improved. Quality control, security control, economic analysis, estimation, and measurement are all topics where additional training is needed for entry-level managers just starting their careers. Table 3 shows the author’s suggested curricula for software project managers:
Table 3: Project Management Courses Days Value 1 Software Milestone Tracking 1.00 10.00 2 Early Sizing Before Requirements 1.00 10.00
3 Software Project Planning 2.00 10.00
4 Optimizing Multi-Country Teams 1.00 10.00 5 Best Practices in Project Management 1.00 10.00
7 Software Cost Estimating: Automated 2.00 10.00 8 Software Security Planning 1.00 10.00 9 Benchmark sources: ISBSG, Namcook. 1.00 10.00 10 Measurement and Metrics of Software 2.00 10.00 11 Software Quality and Defect Estimating 1.00 10.00
12 Software Defect Tracking 1.00 9.75
13 Software Benchmark Overview 1.00 9.75 14 Function Point Analysis: High Speed 1.00 9.75
15 Human Resource Policies 1.00 9.60
16 Software Change Control 1.00 9.50
17 Principles of Software Reuse 1.00 9.40 18 Appraisals and Employee Relations 1.00 9.00
19 Software Cost Tracking 1.00 9.00
20 Software Maintenance & Enhancement 1.00 9.00 21 Methodologies: Agile, RUP, TSP, others 1.00 9.00 22 The Capability Maturity Model (CMMI) 2.00 9.00
23 Tools: Function Points 1.00 9.00
24 Testing for Project Managers 2.00 8.75 25 Static Analysis for Project Managers 0.50 8.75 26 Inspections for Project Managers 0.50 8.75 27 Project Management Body of Knowledge 2.00 8.70 28 Software Metrics for Project Managers 1.00 8.50 29 Software Cost Estimating: Manual 1.00 8.00
30 Tools: Cost Accounting 1.00 8.00
31 Tools: Project management 1.00 8.00
32 Tools: Human Resources 1.00 8.00
33 Tools: Cost and Quality Estimation 1.00 8.00 34 Function Points for Project Managers 0.50 8.00 35 ISO Standards for Functional Measures 1.00 8.00 36 Principles of Agile for Managers 1.00 7.75
37 Principles of RUP for Managers 1.00 7.75 38 Principles of TSP/PSP for Managers 1.00 7.75
39 Earned Value Measurement 1.00 6.75
40 Principles of Balanced Scorecards 1.00 6.50 41 Six-Sigma for Project Managers 2.00 6.00
42 Six-Sigma: Green belt 3.00 6.00
43 Six-Sigma: Black belt 3.00 6.00
TOTAL 36.50 8.74
The total curriculum for software project managers requires more than 36 days and would probably be apportioned over several calendar years. In general software project managers should receive at least five days of training every calendar year.
The History of Software Project Management Tools
Software project management is of course only one of many different forms of project management, such as home construction projects, electrical engineering projects, or civil engineering projects such as building roads and bridges. Software project management does have some distinguishing attributes that seem to differ from other and more established forms of project management:
1. Because software is abstract and invisible to the eye, sizing software projects is more difficult than for many other kinds of project.
2. Because software engineering is not highly automated, manual effort by skilled knowledge workers plays a more important role in software projects than in many other kinds of projects.
3. Because of the large volume of bugs or defects in all software deliverables, quality estimation and quality control are more important for software projects than for many other kinds of project.
4. Because viruses, worms, botnets, and denial of service attacks are endemic and sophisticated, security control is becoming an extremely important aspect of software project management, software development, and software operation.
5. For large software projects in the 10,000 function point range or larger, outright failure, major cost overruns, and major schedule delays occur more frequently than for many other kinds of project.
6. Although changing requirements can occur for any kind of project, they occur with unusual frequency for software projects. Sometimes more than 50% of the delivered features of software projects are not identified until the project is already in process. Software requirements changes have been measured to occur at a rate of between 1% and 2% per calendar month.
7. Some laws such as Sarbanes-Oxley require careful governance of key financial software in large companies, with threats of large fines or criminal charges if governance is absent or ineffective. These new legal liabilities have increased the complexity of management tasks for large projects in large companies.
Long before the computer era and long before software itself, project management was starting to be studied as a technology that needed formalization. Among the pioneers who dealt with project management issues include Henry Gantt whose famous Gantt charts circa 1917 facilitated schedule planning.
Later in the 1950’s Willard Fazar’s “program evaluation and review technique” (PERT) added rigor to isolating and identifying the critical paths which complex projects needed to follow. On the quality side, W. Edwards Deming revolutionized the statistical analysis of quality. Deming’s work was initially adopted in Japan circa 1950 and later came to the United States when it became obvious that quality was a major factor in successful competition. Other pioneers in related topics include Walter Shewart (one of Deming’s teachers), Phil Crosby of ITT, Joseph Juran, and Frederick Taylor.
In terms of software project management tools as opposed to general-purpose project management tools, some of the pioneers include A.J. Albrecht and the invention of function point metrics circa 1973; Dr. Barry Boehm and the development of the famous “constructive cost model” or COCOMO circa 1981; Watts Humphrey and the development of the capability maturity model (CMM) circa 1983; and Peter Hill and the foundation of the International Software Benchmarking Standards Group (ISBSG) circa 1997.
The first software cost estimating tool based on and supporting function point metrics was SPQR/20 released by the author of this paper and Software Productivity Research (SPR) in 1984.
(The acronym SPQR stands for “Software Productivity, Quality, and Reliability. Of course the SPQR acronym is thousands of years old since it was the motto of ancient Rome, “senatus populusque romanus” which means “the senate and the people of Rome.)
The phrase “project management tools” has been applied to a large family of tools whose primary purpose is sophisticated scheduling for projects with hundreds or even thousands of overlapping and partially interdependent tasks. These tools are able to drop down to very detailed task levels, and can even handle the schedules of individual workers. However for software, there are many other critical areas that also need the support of tools: quality prediction and measurement; requirements changes, and the earned-value of ongoing projects. Software projects require additional capabilities to be under full management control.
There are a host of managerial functions that standard project management tools don’t deal with in depth, such as sizing with function point metrics, contract management, personnel management, assessments, quality estimating, defect tracking, and the like.
(Companies such as Namcook Analytics LLC with their Software Risk Master tool have moved into these gaps and produced software sizing and estimating tools that can predict size in terms of 15 metrics, risks, schedules, costs, quality, reliability, and other critical factors.)
The family of project management tools originated in the 1950’s with the U.S. Navy and Air Force. Weapons systems and other complex projects were becoming so large that manual planning and control were no longer feasible. Since computers were starting to become more powerful, it was natural to want to use computer power as an aid for project managers.
Project management tools are an automated form of several management aids developed by the Navy for controlling large and complex weapons systems: the “program evaluation and review technique” (PERT), critical path analysis, resource leveling, and the classic Gantt charts. Such general-purpose tools can of course be used for software projects, but in addition specialized tools are needed that support the unique requirements of software projects.
Project management tools did not originate for software, but rather for handling very complex scheduling situations where hundreds or even thousands of tasks need to be determined and sequenced, and where dependencies such as the completion of a task might affect the start of subsequent tasks. Project management tools are general-purpose in nature, and can be applied to any kind of project: building an aircraft, constructing an office building, or a software project. General-purpose project management tools have no built-in expertise regarding software, as do specialized software cost estimating tools. For example, if you wish to explore the quality and cost impact of an object-oriented programming language such as Objective C a standard project management tool is not the right choice. By contrast, many software cost estimating tools have built-in tables of programming languages and will automatically adjust the estimate based on which language is selected for the application.
Since software cost estimating tools originated about 10 years after commercial project management tools, the developers of software cost estimating tools seldom tried to replicate project management functions such as construction of detailed PERT diagrams or critical path analysis. Instead, the cost estimation tools would export data so that the final schedule could be adjusted via the project management tool. Thus bi-directional interfaces between software cost estimating tools and generic project management tools are now standard features in the commercial estimation market.
Usage Patterns of Software Project Management Tools
Observations made from depositions and testimony during software litigation for breach of contract reveals that software projects that end up in court for outright failure or major overruns usually do not utilize many project management tools. Estimates were often informal and tracking seriously deficient. By contrast, data gathered while performing assessments and benchmarks of successful software projects that combined high productivity and high quality indicate that many specialized project management tools were utilized for both predictions before the projects started and for tracking and measurement while they were underway.
Table 4 shows the approximate sizes of project management tool features measured in function points. The table also illustrates the patterns of tool usage noted on “lagging,” “average,” and “leading” projects. Information on the lagging projects is taken from information discovered during litigation. Information on the average and leading project is taken from benchmark and assessment studies by the author.
Table 4: Patterns of Project Management Tool Usage (Tool size expressed in terms of function point metrics)
Predictive Tools Lagging Average Leading Project sizing (function points) 750 750 Project sizing (lines of code) 250 250 250 Project sizing (requirements changes) 250 Project schedule estimating 1,000 1,500 3,000 Project cost estimating 250 500 3,000
Project quality estimating 2,000
Project risk estimating 2,000
Project value estimating 1,500
Departmental budget planning 2,000 2,000 2,000
SUBTOTAL 3,500 5,000 14,750
Measurement and Tracking Tools Lagging Average Leading
Earned Value Tracking 2,000 3,000
Project cost tracking 2,000 2,000 2,000 Project milestone tracking 1,000 1,500
Project defect tracking 1,500 2,000
Resource tracking 1,000 1,000 2,000 Cost variance reporting 1,000 2,000 2,000
Productivity measurements 1,000
Project benchmark reporting 1,000
Project quality measurement 1,000
SUBTOTAL 4,000 9,500 15,500
TOTAL 6,500 14,500 30,250
Table 4 illustrates two important aspects of software project management tool usage: 1) Leading or sophisticated project managers use more kinds of tools than do managers on lagging and
average projects; 2) Leading or sophisticated project managers use more of the features or select more powerful tools than do project managers on lagging or average projects.
The very significant use of project management tools on leading projects results in one overwhelming advantage: “No surprises.” The number of on-time projects in the leading set is far greater than in the lagging set, and all measurement attributes (quality, schedules, productivity, etc.) are also significantly better.
Software project management tools are far from being a panacea, but advanced knowledge of software project size, potential numbers of defects or bugs, and schedules derived from similar historical projects can go a long way towards yielding a successful conclusion and eliminating the rather high odds of failure associated with large software projects.
For large software projects in large corporations, the project will typically be supported by a “Project Office.” IBM was a pioneer in the utilization of project offices and first deployed them on versions of the main IBM operating system in the early 1970’s.
Typically a project office will be formed for a major software application in the 100,000 function point size range. Applications of this large size have hundreds of development personnel and dozens of subordinate project managers under at least three higher levels of management.
A typical project office for such a large system would contain perhaps half a dozen staff to as many as 10 personnel equipped with a variety of estimating and planning tools, plus historical data from similar projects. The project office would assist the individual project managers in cost estimating, quality estimating, schedule estimating and other predictive tasks. They would also monitor rates of progress, bug reports, earned value, costs, and other status data as the project was underway.
In today’s world of 2013 there are automated tools that provide some support for project offices, such as the “automated project office” or APO tool offered by Computer Aid, Inc.
Recent Evolution of Software Project Management Tools
Project management tools, like other forms of commercial software, continue to evolve and add new features. The following paragraphs summarize some of the recent observations of project management tool capabilities noted in 2008.
Software Project Sizing: Sizing software projects has been a difficult activity for project managers. Sizing in terms of “lines of code” primarily involved either comparison to similar projects or guess work. Adding to the difficulty, there are more than 2,500 programming languages in existence and some of these lack any rules for counting code. Further many software applications utilize several programming languages at the same time. More important, for large software projects the costs and schedules associated with producing paper documents
and finding and repairing defects costs more than the code itself and takes more time during the schedule. These non-coding tasks cannot be estimated or measured using lines of code metrics. Function point metrics have proven to be effective for sizing non-coding work and also programming, so function points are a good choice for life-cycle economic analysis of software projects. Sizing in terms of function point metrics can be done with accuracy from requirements, but by the time requirements are available several iterations of cost estimates are normally required. Also, function point analysis has been somewhat slow and expensive. Certified function point analysts count function points only at a rate of perhaps 400 function points per day. Thus the long schedules and high costs of counting function point metrics have been a barrier
As of 2013 several new forms of high-speed function point tools have been announced. Relativity Technologies has announced a tool for sizing legacy applications called “Function Point Analyzer” or FPA by parsing source code in selected languages. The assertion is that this new tool provides counts with an accuracy equivalent to normal function point analysis, but is able to accomplish this in a matter of minutes rather than a matter of days or weeks. Other companies such as CAST, Relativity Technologies, and Total Metrics also are involved with high-speed function points. There are even open-source function point tools available, such as the “Early and Quick” function point tool available from Italy under a Creative Common license. However an even faster and more versatile high-speed function point sizing method has been patented by the author, Capers Jones, of Namcook Analytics LLC. This method is based on pattern matching and sizes software applications in an average speed of 1.8 minutes for projects between 10 and 300,000 function points. Due to pattern matching, sizing speed is the same for applications of any size. This method is embedded in the Namcook Analytics tool Software Risk Master ™ (SRM).
The Software Risk Master method is metric neutral and supports size in terms of IFPUG function points, logical code statements, and a total of 20 metrics. This method can size both new applications and legacy applications, and operates over a size range that spans from a low of a fraction of 1 function point to a high of more than 300,000 function points. This method also supports the new SNAP size metric for non-function size such as security or performance features. Table 5 illustrates the set of metric supported by Software Risk Master:
Table 5: Size Metrics Supported by Software Risk Master
1. IFPUG function points
2. Non-functional “function points” based on SNAP rules 3. COSMIC function points
4. FISMA function points 5. NESMA function points 6. Simple function points
7. Mark II function points 8. Unadjusted function points 9. Function points “light” 10. Engineering function points 11. Feature points
12. Test-case points 13. Use-case points 14. Story points
15. Lines of code (logical statements) 16. Lines of code (physical lines) 17. RICE objects
18. Physical lines of code 19. Logical code statements 20. Other metrics as needed
Software Risk Master (SRM) supports the largest and most diverse set of size metrics of any predictive tool as of 2013.
The SRM tool is also the world’s fastest sizing method available as of 2013. Table 6 illustrates the sizes of 40 applications as calculated by the Software Risk Master (SRM) high-speed sizing feature. These forty samples took about 75 minutes to produce, or an average of 1.8 minutes per application sized:
Table 6: Examples of Software Size via Pattern Matching Using Software Risk Master ™
Application Size in IFPUG Function Points
1. Oracle 229,434
2. Windows 7 (all features) 202,150
3. Microsoft Windows XP 66,238 4. Google docs 47,668 5. Microsoft Office 2003 33,736 6. F15 avionics/weapons 23,109 7. VA medical records 19,819 8. Apple I Phone 19,366
9. IBM IMS data base 18,558
10. Google search engine 18,640
11. Linux 17,505
12. ITT System 12 switching 17,002
13. Denver Airport luggage (original) 16,661 14. Child Support Payments (state) 12,546
15. Facebook 8,404
16. MapQuest 3,793
18. Android OS (original version) 1,858
19. Microsoft Excel 1,578
20. Garmin GPS navigation (hand held) 1,518
21. Microsoft Word 1,431
22. Mozilla Firefox 1,342
23. Laser printer driver (HP) 1,248
24. Sun Java compiler 1,185
25. Wikipedia 1,142
26. Cochlear implant (embedded) 1,041
27. Microsoft DOS circa 1998 1,022
28. Nintendo Gameboy DS 1,002
29. Casio atomic watch 933
30. Computer BIOS 857
31. SPR KnowledgePlan 883
32. Function Point Workbench 714
33. Norton anti-virus 700
34. SPR SPQR/20 699
35. Golf handicap analysis 662
36. Google Gmail 590
37. Twitter (original circa 2009) 541
38. Freecell computer solitaire 102
39. Software Risk Master™ prototype 38
40. ILOVEYOU computer worm 22
There is also an older and rapid method of function point approximation. The last method, of mathematical conversion between source code and function points has actually existed since 1975. The common name for this method is “backfiring.” The first ratios of source code to function points were developed by A.J. Albrecht and his colleagues at IBM during the development of the original function point metric.
As of 2013 published ratios are available for more than 1,000 programming languages and dialects. However the accuracy of backfiring is low due to variations in code counting methods and variations in individual programming styles. The backfire method is most effective when based on automated counts of logical source code statements, as opposed to counts of physical lines.
A problem with function point sizing as of 2013 is the fact that there are several different kinds of function point metrics. The function point metric defined by the International Function Point Users Group (IFPUG) is the most widely used. But there are also COSMIC function points, Mark II function Points, web-object points, Story points, and several national variants from the Netherlands and Finland, plus usage of mathematical conversion from lines of code to function points called “backfiring.” All of these variants produce different size predications, and as of 2013 there are no proven conversion rules from one variation to another.
(Of course there have been no conversion rules between the variations in counting lines of code, and this problem is now more than 60 years old. About a third of the literature that uses lines of code metrics is based on counts of physical lines; another third is based on counts of logical statements; and the final third reports data in terms of lines of code but fails to identify whether physical or logical lines were the bases of the count. For some languages there can be more than a 500% difference in size based on whether the counts are derived from physical or logical lines of code. Other ambiguous code counting topics include whether data definitions should be included; comments; and blank lines between paragraphs.)
Another problem with function point sizing is that conventional sizing by certified experts is slow and expensive that the practical upper limit is about 10,000 function points. For really massive applications such as large ERP packages and large defense applications that may approach 300,000 function points in size, normal function point analysis would take years. This is one of the economic motives for developing faster and cheaper alternatives to standard function point analysis.
The patented high-speed sizing method embedded in the Software Risk Master tool developed by Namcook Analytics LLC seems to be the world’s fastest as of 2013. It can size software applications in terms of IFPUG function points and logical code statements in about 90 seconds to two minutes, regardless of the overall size of the application. The patented method of operation uses pattern matching to achieve such high speed.
Another new sizing topic getting underway in 2013 deals with the impact of technical and architectural factors on application size. While user-requested functions are measured with function point metrics, part of the size of applications is also due to architectural and technical factors and the “layers” that the application will contain.
In 2011 the International Function Point User’s Group (IFPUG) issued new counting rules for non-functional sizing called “SNAP”. The acronym SNAP stands for “non functional assessment process” more or less.
Other sizing topics are waiting more research and more empirical data: sizing the volume data bases and sizing value other than financial value (i.e. customer satisfaction; enterprise prestige) are two topics that need more research.
In the future a suite of related metrics would allow interesting economic studies of major systems with software, hardware, and data components. The author has suggested a family of 15 functional metrics that would include:
Table 7: Proposed Suite of Functional Metrics for Sizing Complex Systems
1. Application function point metrics 2. SNAP metrics for non-functional size
4. Hardware function point metrics 5. COTS application point metrics 6. Micro function point metrics
7. Data point metrics for data bases and data warehouses 8. Web-site point metrics
9. Software usage point metrics 10. Software learning point metrics 11. Service point metrics
12. Risk point metrics 13. Value point metrics 14. Security point metrics
15. Configuration point metrics (developed by IBM)
This combination of a related family of functional metrics would expand the ability to perform economic studies of modern businesses and government operations that use software, web sites, data, and other business artifacts at the same time for the same ultimate goals.
As an example of these related metrics might be used, consider a complex system such as new kind smart appliance:
• 1,000 software function points at a cost of $750 each • 200 SNAP points at a cost of $75 each
• 2,000 hardware function points at a cost of $1,500 each • 5,000 data points at a cost of $500 each
• 500 risk points with a liability of $10,000 each
• 25,000 value points with a potential value of $20,000 each
Many modern devices use combinations of software, hardware, and data. They have potential risks and they have potential value. A suite of related metrics using the logic of function points would allow integrated sizing and estimating of complex hybrid systems that encompass hardware and data as well as software.
Software Project Schedule Planning: There are dozens of commercial tools, in-house tools, and even open-source freeware tools that can perform critical path analysis of software projects, as well as other kinds of projects. Examples of such tools include Microsoft Project, Artemis Views, and TimeLine. However for software projects schedule planning can also be performed by specialized software estimating tools such as COCOMO II, CostXpert, KnowledgePlan, Price-S, SEER, SLIM, and Software Risk Master (SRM) in alphabetical order.
The Spanish company OptiMyth has recently released a new kind of static analysis tool that includes the interesting feature of predicting the hours of effort needed to repair every bug that the tool identifies.
Overall a detailed work-breakdown structure that reaches the level of specific tasks and individual assignments is the most effective, although complex and not easy to achieve. This level of granularity can only be provided by automated tools: manual methods are too labor-intensive and also to difficult to update.
Manual schedule estimates are notoriously unreliable above 250 function points. Parametric estimation tools such as Software Risk Master hold their accuracy up above 100,000 function points.
As of 2013, schedule estimating remains a difficult challenge for software project managers. Above 10,000 function points in size, about two thirds of all software projects run late, exceed their budgets, or are cancelled because the delays and overruns have degraded the return on investment to a negative number. Automated parametric tools are far more accurate for large applications than manual estimates.
Two critical factors tend to make software schedules longer than initially anticipated. One of these factors is growth of creeping requirements, which averages more than 1% per calendar month. The second is inadequate defect prevention combined with failure to use early defect removal activities such as inspections or static analysis. Detailed study of software slippage indicates that testing is the main activity where schedule slippage tends to be severe.
Software Cost Estimating: Software cost estimation can be traced back to the first IBM software cost estimating tool in 1973 and also to the first commercial estimating tool which was the Price-S tool originally marketed by the RCA corporation. These pioneering tools were soon followed by other commercial software cost estimating tools such as Checkpoint, COCOMO, Estimacs, ISBSG, SEER, SLIM, SoftCost, SPQR/20, Software Risk Master (SRM) and others. These specialized software cost estimating tools were often superior to manual cost estimates, and they also included schedule and quality estimates as well. Because the software estimating tools appeared about 10 years after general-purpose project management tools, they support bi-directional data transfer with general purpose tools such as Microsoft Project or Artemis Views. Another advantage of the commercial software cost estimating tools is that they contain built-in knowledge of topics that affect software projects, which are lacking in general-purpose tools. Examples of these factors include the specific programming languages to be used, the capability maturity level integration CMMI level of the development organization, the methodology to be used such as Agile development, Rational Unified Process (RUP), Team Software Programming (TSP), or Extreme programming, and the structure or complexity of the application.
The commercial software estimating tools also have built-in knowledge of key topics such as the defect removal efficiency levels of various kinds of inspection and testing. The first commercial estimating tool that could predict defect quantities and defect removal efficiency levels was SPQR/20, which entered the market in 1984. This was also the first commercial estimating tool based on function point metrics, although it predicted lines of code as well.
Large software projects often spend more money on the production of paper documents such as requirements, specifications, plans, user manuals, etc. than they do on the production of the source code itself. Estimating these costs lagged for many years because lines of code metrics could neither measure nor predict paperwork. However modern commercial software estimating tools can predict the sizes and costs for more than 50 kinds of paper document, which collectively can accumulate more than 35% of the total development cost of a large software project. A few software cost estimating tools can even predict the costs of Nationalization, or converting some documents into foreign languages for international sales.
The Software Risk Master (SRM) tool of Namcook Analytics LLC predicts document sizes, completeness, and costs for more than 25 kinds of software documents including requirements, architecture, design, user manuals, training materials, etc. Because of the patented SRM early sizing feature, this tool is the only tool that can predict requirements size before the requirements are completed. The tool also predicts requirements defects and “toxic requirements” that should not be in the software at. The famous “Y2K” problem is an example of a toxic requirement.
Software Project Risk Estimating: The major forms of risk for software projects include outright cancellation due to failure to reach operational stability, cost overruns, schedule overruns, poor quality, security vulnerabilities, difficult human interfaces, and sluggish performance. Since software project failures are common, it can be stated that software risk assessment and risk predictions are either inadequate or not performed with rigor. There are also business risks such as indictment under the Sarbanes-Oxley law for inadequate governance of key financial applications. In the modern world there are also increasing severe risks from hackers, viruses, and denial of service attacks.
As of 2013 there are some risk prediction features in commercial estimating tools such as KnowledgePlan, SLIM, SEER, and Software Risk Master (SRM). There are also some stand-alone risk models as well as risk estimation in integrated tools such as the Automated Project Office tool (APO) by Computer Aid Inc.
However the risk analysis features of Namcook Analytics’ Software Risk Master tool are the most comprehensive. Table 8 shows the full suite of 210 risks that Namcook’s tool can analyze:
Table 8: Risks Analyzed by Software Risk Master Health and Safety Risks
2 Risk of application failure causing serious illness or injuries 3 Risk of application violating FDA or other regulations 4 Risk of application failure damaging medical activates 5 Risk of application causing environmental damages 6 Risk of team fatigue due to excessive overtime
Security Risks
7 Risk of loss or theft of proprietary source code 8 Risk of electromagnetic pulse shutting down software 9 Risk of application failure degrading national security 10 Risk of data theft from application
11 Risk of physical security breach at software locations 12 Risk of poor security flaw removal
13 Risk of security flaws in application 14 Risk of poor security flaw prevention
15 Risk of violating the 25 SANS coding problems 16 Risk of security flaws in uncertified reused code
17 Risk of deliberate "back door" traps placed by developers 18 Risk of theft of intellectual property
Quality Risks
19 Risk of excessive defect levels: > 6.0 per function point 20 Risk of defect removal efficiency < 85%
21 Risk of poor data quality with serious errors
22 Risk of inadequate defect removal methods, low efficiency 23 Risk of poor estimation of bugs, defect removal efficiency 24 Risk of premature application release with excessive bugs 25 Risk of not using pre-test inspections: requirements, design 26 Risk of poor test case and test script design methods 27 Risk of poor test library controls
28 Risk of testing with amateurs rather than professionals 29 Risk of high code complexity that raises "bad fixes" > 10% 30 Risk of error-prone modules in application
31 Risk of claiming to use inspections, but only partially 32 Risk of late and inadequate defect tracking
33 Risk of poor test coverage
34 Risk of poor quality in COTS packages
35 Risk of insufficient Quality Assurance (QA) reviews 36 Risk of poor quality in reused components
37 Risk of not using pre-test static analysis of source code 38 Risk of understaffing Quality Assurance
39 Risk of poor quality in outsourced projects 40 Risk of errors or bugs in test cases 41 Risk of low operational reliability
42 Risk of poor quality by open-source providers 43 Risk of duplicate test cases
Legal Risks
44 Risks of patent litigation from competitors
46 Risk of inadequate warranties for quality and security 47 Risk of Sarbanes Oxley litigation
48 Risk of incurring contract penalties
49 Risk of poorly constructed contracts that leave out risks 50 Risk of poorly constructed contracts that leave out quality 51 Risk of former employees violating non-compete agreements 52 Risk of breach of contract litigation on outsourced projects 53 Risk of application failure causing violations of laws 54 Risk of "cease and desist" warnings of alleged patent flaws
Traditional Software Risks
55 Risk of toxic requirements that should be avoided 56 Risk of inadequate progress tracking
57 Risk of development tasks interfering with maintenance 58 Risk of maintenance tasks interfering with development 59 Risk that designs are not kept updated after release 60 Risk of unstable user requirements growth >1% per month 61 Risk that requirements are not kept updated after release 62 Risk of clients forcing arbitrary schedules on team 63 Risk of omitting formal architecture for large systems 64 Risk of inadequate change control
65 Risk of executives forcing arbitrary schedules on team 66 Risk of not using a project office for large applications 67 Risk of missing requirements from legacy applications 68 Risk of missing user requirements due to user uncertainty 69 Risk of slow application response times
70 Risk of inadequate maintenance tools and workbenches 71 Risk of application performance problems
72 Risk of poor support by open-source providers
73 Risk of reusing code without test cases or related materials 74 Risk of excessive feature "bloat"
75 Risk of inadequate development tools
76 Risk of poor help screens and poor user manuals 77 Risk of slow customer support
78 Risk of inadequate functionality Financial Risks
79 Risk of application failure causing major financial loss 80 Risk of consequential damages > $1,000,000,000 81 Risk of project termination due to poor quality, overruns 82 Risk of features slipping from planned release
83 Risk of significant project cost overruns
84 Risk of project value dipping below project costs 85 Risk of "leakage" from software cost and historical data 86 Risk of bankruptcy by vendor
87 Risk of negative earned value for project 88 Risk of significant project schedule overruns
89 Risk of application failure causing moderate financial loss 90 Risk of cost overruns on outsourced projects
92 Risk of unbudgeted costs from security attacks 93 Risk of unbudgeted costs from litigation 94 Risk of inadequate cost accounting 95 Risk of bankruptcy by client
96 Risk of application violating standard accounting practices Business Risks
97 Risk of missing critical market window 98 Risk of losing clients due to faulty software 99 Risk of application failure damaging business data 100 Risk of application failure damaging distribution 101 Risk of application failure damaging transportation 102 Risk of application failure affecting operation of equipment 103 Risk of application failure damaging retail activities 104 Risk of competitive applications with better features 105 Risk of application obsolescence before completion 106 Risk of application failure damaging law enforcement 107 Risk of application failure damaging government activities 108 Risk of application failure damaging communications 109 Risk of poor governance by executives
110 Risk of application failure damaging manufacturing 111 Risk of application failure damaging stock values 112 Risk of application failure shutting down vital equipment 113 Risk of rubber-stamp phase reviews without real oversight 114 Risk of poor or missing project historical data
115 Risk of executive and client dissatisfaction with project 116 Risk of poor support by COTS vendors
117 Risk of cost and schedule overruns for ERP deployment Social Risks
118 Risk of significant layoffs of project team
119 Risk of poor managers driving out top technical staff 120 Risk of termination for cause of key technical personnel 121 Risk of ignoring learning curve for new methodologies 121 Risk of voluntary attrition of key technical personnel 122 Risk of poorly planned organization structures 123 Risk of team sizes too small for application 124 Risk of too few personnel per manager (< 5) 125 Risk of too many personnel per manager (> 15) 126 Risk of poor organization structures
127 Risk of low team morale from excessive schedule pressure 128 Risk of poor communication among supply chain members 129 Risk of stakeholder disputes that change requirements 130 Risk of poor communications among team members 131 Risk of team size too large for application
132 Risk of low user satisfaction levels
133 Risk of inadequate user training for complex new software 134 Risk of poor communications with stakeholders
135 Risk of major management disagreements 136 Risk of strikes by unionized personnel
External Risks
137 Risk of natural disaster affecting projects
138 Risk of loss of stakeholders or clients during development 139 Risk of accidental loss of key personnel during development 140 Risk of serious medical problems of key personnel
Ethical Risks
141 Risk of fraudulent progress/status reports 142 Risk of project managers ignoring risks
143 Risk of project managers concealing risks from clients 144 Risk of non-compete violations by former employees 145 Risk of false claims by methodology enthusiasts 146 Risk of false claims of high CMMI levels
147 Risk of claiming to use a methodology, but not really doing so 148 Risk of false claims by outsource vendors
149 Risk of false claims by COTS vendors Knowledge Risks
150 Risk of users not fully understanding their own requirements 151 Risk of inadequate requirements analysis for large systems 152
Risk of adopting new methods as a cult rather than a technology
153 Risk of effective solutions not being known by managers 154 Risk of effective solutions not being known by team 155 Risk of inadequate sizing prior to funding project 156 Risk of inadequate schedule planning
157 Risk of late start in deploying risk solutions 158 Risk of manual estimates for large applications 159 Risk of excessive optimism in initial plans, estimates 160 Risk of estimates being rejected due to lack of benchmarks 161 Risk of using unsuitable development methodology 162 Risk of poor project oversight by clients
163 Risk of "good enough" fallacy applied to application 164 Risk of insufficient project management skills 165 Risk of poorly trained management personnel 166 Risk of inadequate user-error prevention 167 Risk of team skills not matching project needs 168 Risk of poorly trained maintenance personnel 169 Risk of inadequate defect prevention
170 Risk of poorly trained development personnel 171 Risk of poorly trained support personnel 172 Risk of poorly trained test personnel 173 Risk of application architectural flaws
174 Risk of inadequate user guides and HELP screens 175 Risk of poor usability and poor interfaces 176 Risk of poorly trained QA personnel
177 Risk of international misunderstandings for global projects 178 Risk of inadequate programming languages
180 Risk of application violating international standards Enterprise Risks
181 Risks of obsolete or cumbersome enterprise architecture 182 Risks of difficult data migration from legacy applications 183 Risk of merger or takeover causing layoffs and cancellations 184 Risks from disconnected "stove pipe" applications
185 Risks of fragmented software ownership in enterprise 186 Risks of uncoordinated redundant applications and data 187 Risks of uncoordinated multi-national development groups
Merger, Acquisition, and Venture Capital Risks
188 Risk of dilution of ownership due to multiple funding rounds 189 Risk of cutting R&D after mergers or venture investment 190 Risk of losing key personnel after mergers.
191 Risk of bankruptcy within three years of venture funding 192 Risk of inadequate due diligence prior to completion 193 Risk of eventual competition by dissatisfied personnel 194 Risk of poor integration of teams after mergers
195 Risk of venture-backed boards eliminating key technical staff 196 Risk of venture-backed boards damaging business prospects 197 Risk of bankruptcy within three years of mergers
Technology Risks
198 Risk of hardware changes making application obsolete 199 Risk of hardware changes requiring extensive rework 200 Risk of related software changes requiring rework 201 Risk of supply chain changes requiring rework 202 Risk of standards changes requiring rework
203 Risk of directive to adopt multiple hardware platforms 204 Risk of withdrawal of hardware/software platforms 205 Risk of unplanned directive to adopt multiple software platforms
Embedded Software Risks
206 Risk of software problems raising hardware liabilities 207 Risk of software problems causing unrepairable failures 208 Risk of software problems causing patent violations 209 Risk of software problems delaying hardware products 210 Risk of software raising hardware warranty costs
Research and prototypes of risk analysis tools are a sign that future kinds of risk analysis may improve the situation. In particular, security vulnerabilities need better predictive capabilities in the modern world where viruses, spyware, and denial of service attacks are daily events that affect thousands of companies and millions of software applications. As outsourcing becomes more and more common, litigation avoidance is also a risk that needs more attention than it has received to date.
Another emerging kind of risk prediction is that of predicting the odds of litigation for contract projects. Indeed there is a working prototype of a litigation risk tool that also predicts the probable costs for the plaintiff and defendant in breach of contract litigation. This is a standard feature in Software Risk Master (SRM).
Software Quality Estimating: Quality estimation for software was not well done before the invention of function point metrics because lines of code could not be used for either measuring or predicting defects in requirements, design, user documents and other non-code items. Once function point metrics were applied to the task of quality prediction, it was soon discovered that errors in requirements and design outnumbered coding errors. Commercial software estimating tools that could predict the full spectrum of software defects from all major sources (requirements, design, code, user documents, and “bad fixes” or secondary defects) only became available in 1985 with the SPQR/20 commercial estimation tool. As of 2013 the Software Risk Master tool can predict software defects found in requirements, design, code, user manuals, and “bad fixes” or secondary defects.
However in today’s world of cloud computing, Software as a Service (SaaS), open-source components, and commercial off-the-shelf (COTS) packages quality estimation needs to expand beyond a single application. Quality estimation needs to encompass all components of modern systems, both in-house and acquired from external sources. This is an area that needs more data and additional research as of 2013.
Another important aspect of quality estimation is that of defect removal efficiency, or predicting the percentages of defects that will be found by various kinds of review, inspection, and test stage. Historical data gathered since the 1970’s indicate that formal design and code inspections have the highest defect removal efficiency and alone can top 85%. Static analysis tools also can top 85% in terms of defect removal efficiency. However static analysis tools as of 2012 only support about 25 programming languages out of more than 2,500 that are known to exist. Most forms of testing such as unit test, new function test, regression test, etc. are less than 35% efficient or only find about one bug out of three. Some of the newer forms of testing such as automated unit test can approach static analysis or top 50% in defect removal efficiency. A sequence of formal requirement and design inspections, followed by static analysis of code, followed by six to eight different testing stages can top 98% in cumulative defect removal efficiency. Since the industry average for defect removal efficiency has stayed almost constant about 85% for more than 30 years, companies and projects that top 95% are to commended. As of 2013 several commercial estimating tools can predict defect potentials and defect removal efficiency levels.
Some commercial estimating tools can also predict the specific sequence of inspections, static analysis, and test stages that should be used to achieve optimal levels of defect removal efficiency. Numbers of test cases and test coverage can also be predicted for many forms of
testing such as unit test (manual or automated), regression test, new function test, performance test, system test, etc.
Since most forms of testing are less than 35% efficient in finding bugs, it is obvious that a multi-stage series of inspections, static analysis, and tests will be needed to achieve high levels of defect removal efficiency. The good news about this fact is that projects that approximate 95% in cumulative defect removal efficiency have shorter schedules and lower costs than “average” projects with only about 85% cumulative defect removal efficiency. Maintenance costs are also reduced. Phil Crosby’s aphorism that “Quality is Free” has not only been proven, but also measured on many successful software projects.
Because the costs and schedules associated with finding and fixing bugs is the most significant cost element of large software projects and the primary factor leading to cancellation and delay, quality prediction is a critical element of successful cost and schedule prediction. Poor quality is a frequent cause of litigation too.
Many specific software quality methods such as six-sigma for software, quality function deployment (QFD), formal design and code inspections, static analysis, automated testing, error-prone module removal, and joint application design (JAD) can now be modeled and predicted using automated tools. The most interesting results come from side-by-side comparisons that show a specific approach such as six-sigma compared against an identical project that did not use the same approach.
Table 9 illustrates a “best case” quality prediction from the Software Risk Master (SRM) tool for an application of 1000 function points using the Java languages, and developed by an experience team at CMMI level 5:
Table 9: Sample Quality Predictions from Software Risk Master
OUTPUT DATA
Requirements defect potential 134
Design defect potential 561
Code defect potential 887
Document defect potential 135
Total Defect Potential 1,717
Per function point 1.72
Per KLOC 32.20
Defect Prevention Efficiency Remainder Bad Fixes
1 JAD 27% 1,262 5
2 QFD 30% 888 4
4 Models 68% 229 5
Subtotal 86% 234 15
Pre-Test Removal Efficiency Remainder Bad Fixes
1 Desk check 27% 171 2
2 Static analysis 55% 78 1
3 Inspections 93% 5 0
Subtotal 98% 6 3
Test Removal Efficiency Remainder Bad Fixes
1 Unit 32% 4 0 2 Function 35% 2 0 3 Regression 14% 2 0 4 Component 32% 1 0 5 Perfomance 14% 1 0 6 System 36% 1 0 7 Acceptance 17% 1 0 Subtotal 87% 1 0 PRE-RELEASE COSTS 1,734 3
POST-RELEASE REPAIRS (TECHNICALDEBT) 1 0
MAINTENANCE OVERHEAD
COST OF QUALITY (COQ)
Defects delivered 1 High severity 0 Security flaws 0
High severity % 11.58%
Delivered Per FP 0.001
High severity per FP 0.000
Security flaws per FP 0.000
Delivered Per KLOC 0.014
High severity per KLOC 0.002
Security flaws per KLOC 0.001
Cumulative Defect 99.96%
Software quality predictions are critical for successful project management because for more than 50 years the costs of finding and fixing defects or bugs has been the Number 1 cost driver for the entire software industry.
A combination of defect prevention, pre-test defect removal such as inspections and static analysis, and formal mathematically based testing are key factors for success in building large software applications.
Software Project Outsource Contract Analysis: Since many outsource contracts are cancelled and about 5% end up in court for breach of contract, it can be stated that the available tools for predicting the results of software outsource contracts need improvement. There are some specialized and proprietary tools available from consulting companies that assist clients in dealing with outsource vendors. Also, normal commercial software cost estimating tools can be us
