A Fool with a Tool:
Improving Software Cost and
Schedule Estimation
Ian Brown, CFPS Booz Allen Hamilton
2006 International Software Measurement and Analysis Conference
A fool with a tool
is still a fool.
Agenda
Why Estimate?
The Challenges of Software Estimation
Benefits of Using a Tool
Pitfalls of Using a Tool
Why do we need to estimate?
Because we need to know how much to budget or to bid
Because resources (people) are scarce and we need to know how to allocate them in the best way
Because we need to better predict what it will take the next time
The purpose of this presentation is to discuss the benefits and pitfalls of
software estimation tools as well as other key estimation best practices
0 2 4 6 8 10 12 2-05 4-05 6-05 8-05 10-05 12-05 2-06 4-06 6-06 Months Staff Level
Estimate: 22244 Hours; 16.11 Months
Project Staffing Plan
0 2 4 6 8 10 12 2-05 4-05 6-05 8-05 10-05 12-05 2-06 4-06 6-06 Months Staff Level
Estimate: 22244 Hours; 16.11 Months
Project Staffing Plan
Software development projects are notorious for cost and
schedule overruns
Source: The Standish Group, CHAOS
Chronicles Version 3.0, 2002 Succeeded 34% Challenged 51% Failed 15%
Challenged Projects by Schedule Overruns
30% 29% 26% 14% 1% 0% 10% 20% 30% 40% <20% 21-50% 51-100% 101-200% >200%
Project Resolution by Type
Challenged Projects by Cost Overruns
46% 27% 18% 7% 2% 0% 10% 20% 30% 40% 50% <20% 21-50% 51-100% 101-200% >200%
Challenges of Software Estimation
Some of this problem is due to inaccurate or optimistic cost and
schedule estimates up front
Why is software estimation so difficult?
Dynamic environment– Changing languages and technologies
– Changing requirements
Often treated as a “side show”
Lack of relevant historical data
Expert judgment approach (and other manual estimation techniques) not usually reliable
Complex activities not always factored
Challenges of estimating software size
Challenges of Software Estimation
Would you want this guy doing your
Automated COTS estimation tools leverage large historical data
sets and flexible input parameters
Examples
Examples
Tools are parametric in nature, meaning calculations are based on complex statistical algorithms
Outputs from model are based on input assumptions
– Size
– Personnel skills and experience
– Development environment
– Productivity factors
– Labor rates
Estimates can be generated with as much or as little information as is available
Tools typically estimate all development life cycle activities, including various levels of testing
Tools can handle the complex factors that can impact cost and schedule on a software project
The same flexibility that makes estimation tools so useful also
provides the opportunity for real problems
Estimation tools can produce virtually any number you want them to produce
Estimated activities from the tool may not match up with expected activities on the project
The path to the Dark Side of estimation tools:
– “Let’s just assume a better productivity factor.”
– “I’m sure we’ll be able to reuse most of the software.”
– “That estimate does not fit the project budget. We need to change the parameter input assumptions.”
– “It’s what the tool says, so it must be right.”
Pitfalls of Tools
Use an estimation tool – and understand the potential pitfalls
The benefits of using a parametric estimation clearly outweigh the pitfalls
– If you are aware of the potential dangers, you can avoid them
Software development is an inherently complex activity that can be influenced in many ways by many different factors
– Manual estimation methods simply cannot account for all the possible factors and impacts
– A tool provides the ability to consider these factors in the estimate and to conduct “what if” scenarios or sensitivity analysis fairly easily
It’s All About the Process
In order to improve, you first must understand where you are
In order to understand where you are, you need to set a baseline
In order to baseline, you should have a repeatable, standard process
The process should be documented and followed
Improving Software Estimation
Develop work breakdown structure (WBS) Develop work breakdown structure (WBS) Estimate software size Estimate software size Establish key project parameters Establish key project parameters Assess detailed project parameters Assess detailed project parameters Develop and refine cost model Develop and refine cost model Perform risk assessment and sensitivity analysis Perform risk assessment and sensitivity analysis Crosscheck and validate estimates Crosscheck and validate estimates Develop work breakdown structure (WBS) Develop work breakdown structure (WBS) Estimate software size Estimate software size Establish key project parameters Establish key project parameters Assess detailed project parameters Assess detailed project parameters Develop and refine cost model Develop and refine cost model Perform risk assessment and sensitivity analysis Perform risk assessment and sensitivity analysis Crosscheck and validate estimates Crosscheck and validate estimates c d e f g h i Develop work breakdown structure (WBS) Develop work breakdown structure (WBS) Estimate software size Estimate software size Establish key project parameters Establish key project parameters Assess detailed project parameters Assess detailed project parameters Develop and refine cost model Develop and refine cost model Perform risk assessment and sensitivity analysis Perform risk assessment and sensitivity analysis Crosscheck and validate estimates Crosscheck and validate estimates Develop work breakdown structure (WBS) Develop work breakdown structure (WBS) Estimate software size Estimate software size Establish key project parameters Establish key project parameters Assess detailed project parameters Assess detailed project parameters Develop and refine cost model Develop and refine cost model Perform risk assessment and sensitivity analysis Perform risk assessment and sensitivity analysis Crosscheck and validate estimates Crosscheck and validate estimates c d e f g h i
If an estimation process is not defined and
followed, it is impossible to tell where
Measure the Process
Baseline the current estimation process: track estimates and collect actual data once project is complete
Set improvement goals and establish measurement processes to ensure information is collected and analyzed
Improving Software Estimation
Root cause analysis and “detective work” will
identify weak areas in the estimation process and may suggest possible improvements
Improve Estimation Capability
How accurate are our schedule estimates?
How accurate are our effort estimates?
How accurate are our cost estimates?
How accurate are our size estimates? Estimated Schedule Actual Schedule Estimated Effort Actual Effort Estimated Cost Actual Cost Estimated Size Actual Size Goal: Questions: Measures: Schedule Variance Indicators: Size Variance Cost Variance Effort Variance
Size Matters
Size is the most critical driver of cost and schedule on a software project.
– All other things equal, the larger the size, the greater the effort and cost and the longer the schedule
Improving Software Estimation
Same design, same construction, but the first is
twice the square feet – which house will cost more
Size Matters (cont’d)
Having a consistent, well-defined, standard size measure as part of the estimation process is critical
Improving Software Estimation
Use IFPUG function points!
Application Being AssessedInternal Data Inputs Inputs Outputs Outputs Inquiries Other Applications/Systems External Interfaces End user Inputs Outputs Inquiries
Recent pilot study in Booz Allen to evaluate function points versus objects as a size measure
Size Matters (cont’d)
Improving Software Estimation
Variance Actual Cost Estimated Cost Function Points Release $205,703 $175,534 $361,537 $204,800 $162,377 $359,500 CAERS 3.0 It 2 CAERS 3.0 It 1 CAERS 2.07 7.50% 241 0.43% 302 0.56% 652 Variance Actual Cost Estimated Cost Function Points Release $205,703 $175,534 $361,537 $204,800 $162,377 $359,500 CAERS 3.0 It 2 CAERS 3.0 It 1 CAERS 2.07 7.50% 241 0.43% 302 0.56% 652
Average FP estimation variance of 2.83% demonstrates highly reliable estimation methodology
Correlation between FP size and actual cost: 0.9977
– This strong correlation indicates a strong predictive relationship between FPs and actual cost
– It also demonstrates the consistency of the FP sizing methodology over multiple releases
Track 1 Project Team Estimate size based on object definition Estimate cost based on past experience Track 2 CFPS Estimate size based on function points
Estimate cost with parametric tool
Analyze estimates versus actuals
Make sure that estimated activities from the tool match up with
actual activities on the project
Some activities on the project may not be included in the estimate produced by the tool
– Training
– Installation
– Data cleansing, etc.
Improving Software Estimation
Some activities in the estimate may not be part of your project
– Example: Project contract is for requirements through program test. System test and end user acceptance test are outside the contract scope. Be sure to back this cost and schedule out of the estimate produced by the tool
Re-estimate throughout the project life cycle
Project scope and requirements will change – cost and schedule
estimates should be reviewed and revised to reflect these changes
Volatility and scope creep may be factored into original estimate, but the cost, schedule, and size
estimates should still be reviewed as part of the configuration control process to ensure the project is still on track
Improving Software Estimation
Revised Cost and Schedule Estimates Revised Cost and Schedule Estimates
Decide which changes to include in scope of
project
Decide which changes to include in scope of project Client Client Project Manager Project Manager Project Team Project Team Assess Impact to Cost and Schedule
Assess Impact to Cost and Schedule
Change Requests New Requirements Redefined Requirements
Change Requests New Requirements
Redefined Requirements Project TeamProject Team
Client Client System Experts System Experts Baseline Cost and Schedule Estimates Baseline Cost and Schedule Estimates Function Point Analysis
and Parametric Model
Function Point Analysis and Parametric Model
Project Manager Project Manager Documentation Documentation REPEAT REPEAT
Don’t assume that the tool is necessarily right
Review the estimates in the context of the project and other benchmarks. Answering these questions helps to increase the confidence in the estimates
– Are the estimates consistent with other projects or similar size and nature?
– Are the estimates consistent with previous organizational experience?
– How does the projected productivity compare to industry best practice standards?
– Does the projected staffing plan make sense?
Develop a crosscheck estimate with another method or tool
– If the estimates are wildly different, review and reconcile
Conduct post-project reviews (PPR)
PPRs are critical to improving both the estimation process and the accuracy of future estimates
Collect lessons learned, identify weak areas, and suggest potential improvements
Compare actuals to estimates (both initial and final estimates) and analyze variance
– Check size for accuracy
– Review parameter assumptions
– Review project activities
– Calibrate model if appropriate
OK, I’ve heard your little speech – now what?
Estimation does not have to be a guessing game
Leverage the power of estimation tools, but
understand the nuances and potential stumbling blocks of doing so
Establish an estimation process – document the steps and follow them
Use function points to size your software as a component of the estimation process
Measure - baseline the environment and track improvement over time
