GETTING THE BIGGEST BANG FOR YOUR
SOURCE CODE ANALYSIS BUCK
Combine a smart deployment strategy with realistic goals,
and you’ll achieve your payback in no time
Companies that create smartphones, military systems, aerospace technology, medical devices, and
communications software and equipment are all looking at source code analysis (SCA) as a way to
reduce their costs while creating more secure and reliable code. Naturally, people want to know
what payoff to expect from deploying SCA and how they can show ROI within their organization.
This paper shows you how to build a business case for source code analysis and demonstrates a
few different ways to calculate ROI for this technology. Key topics covered in this paper include:
• Fundamental ROI principles for source code analysis
• Three areas that can ruin your business case before you get started
• Example ROI model that you can use in your own deployment
As every software development professional knows, the later bugs are found, the more they cost to fix. This make sense, of course. But unfortunately, as the chart below shows, while the majority of bugs are introduced during the coding phase of the development process, most are not actually detected and fixed until the later — and more expensive — stages. To improve efficiency and save money, development teams should be moving a larger percentage of that bug detection and repair into the coding stage, where it’s less expensive to deal with. If you take that approach, you’ll get a big payoff, as long as the approach you introduce doesn’t add more cost or effort to the front end of your process. And that’s really the primary value proposition for the latest generation of source code analysis tools. Designed to find critical defects and security vulnerabilities as the developer is writing their code (following the usability model of a spell checker), errors can be found and fixed the instant they’re created. Source code analysis is automated and requires no significant amount of effort by the developer, other than reviewing and correcting defects identified by the tool.
Figure 1 | Defect Detection vs. Cost to Repair Source: Capers Jones
Before you present this chart as a way to justify your investment in SCA, let’s take a look at some best deployment practices to protect you from common pitfalls that can imperil your business case (see 3 Deployment Rules for ROI Success below). Once armed with these deployment do’s and don’ts, we provide you with specific tips for building an
ROI model that works for your organization (see How to Build a Business Case and Show ROI).
3 DEPLOYMENT RULES FOR ROI SUCCESS
All the detailed ROI numbers and cost justifications are great, but if you don’t follow the rules outlined below, you’ll have a tough time achieving the business goals and ROI model you’ve put in place.
1. Don’t Fall into the “Smoking Gun” Trap
As a development team, you need to be in agreement that the source code analysis tool is finding real and important issues that need to be addressed. This is really the fundamental assumption that underlies your adoption of the technology in the first place, and flows into your ROI model.
The mistake many engineering teams make is they instinctively want to justify their source code analysis deployment by finding that one EPIC & LEGENDARY bug that could or did cost them millions of dollars in the field. We call this the Smoking Gun bug, and if you find it, you can stop reading this paper because you have your cost justification. The reality is that most deployments have trouble correlating their source code analysis results to the Smoking Gun bug, if for no other reason than the complexity of doing so. You need to correlate hundreds of bug reports that were found statically with debug information from this single colossal field failure, and prove that it would have been found with a source
The true value of SCA isn’t about the Smoking Gun; it’s about the cumulative effect of reducing hundreds, and more likely many thousands of bugs from your code with each and every build, story, and release.
We see this time and time again when customers stop new issues from being introduced, systematically burn down their backlog of Klocwork-reported bugs, and measure the results. From a Polycom case study:
[...]The impact of SCA on the HDX product line can be measured in customer satisfaction and product stability. “We are getting fewer customer call-ins, and we see better stability in test cycles.” That improvement can be measured empirically through Polycom’s field data. “Our service cases have reduced significantly and that is trending positive. Our customers are highly satisfied.”
To appreciate the cumulative effect, you need to satisfy yourself and the development team that, across the board, the tool is finding real and important issues that must be fixed.
The most common approach to achieving this goal is to have a few key developers who are respected and have good knowledge of the code review a set of bug reports generated by the source code analysis tool — normally from a code base that’s already been through your normal testing process — and assess their validity. This is a simple, cost-effective way to achieve group-level confidence in the tool.
A novel approach to achieving this goal was conducted by Overture Networks, as described below:
Each time a detected bug was fixed, the developer included a log statement in the revised code. During lab testing, if the fixed code ran, the benign statement would appear to let the test team know that, “If we hadn’t made Bug Fix X, Error Y would have occurred here.” These instances were documented by the test lab.
Regardless of how you do this, once you’ve proven that the tool is finding real issues, you can move on to more productive activities.
2. Avoid Hidden Deployment Costs
Let’s start by stating the obvious: You’ll reduce your ROI if you increase your costs. So, how do you avoid this situation? There are two cost-chewing activities that every source code analysis deployment can run into: False Positives and Opening the Flood Gates.
The Cost of False Positives
In the case of false positives, you can run into a couple of quagmires, both of which are very avoidable:
• The first is selecting a tool that, frankly, isn’t very good. There are some source code analysis tools out there that throw 99% false positives, and the defects they do find aren’t that interesting (see Rule #1). To avoid this trap, evaluate modern source code analysis tools that run a comprehensive build of your code, perform deep, interprocedural dataflow analysis, and use some kind of formalism in their analysis, such as symbolic logic. If you’re dealing with a tool that doesn’t meet these criteria, don’t bother.
• The other false positive quagmire is what we call “perceptual” false positives. These are situations where the tool is correct but developers provide the following reaction: “Meh, no big deal.” This is directly related to the issues discussed in Rule #1 (Smoking Gun) but also goes to a bigger deployment issue that you need to address. Developers need to be on-board, understand the value, and realize the cumulative effect that cleaning
thousands of issues will have on their code over time. It will save them time, will help stop the repetitive and distracting process of reworking issues found by testing or in the field, and will generally make them better developers. It’s not as glamorous as finding the Smoking Gun bug, but it works.
Opening the Flood Gates
There are two sure-fire ways you can blunder into this mistake:
• Target all your product lines/teams at once. You need to work through your engineering teams and/or
product lines in a systematic way. Pick a product line or engineering team — some sort of cohesive group of people and code that makes sense in your organization — and start your source code analysis deployment there. Once you’ve done a rollout with this group and worked through the inevitable lessons-learned and developed some key best practices that work for your organization, then you can begin expanding the SCA deployment to other groups.
• Deal with backlog on Day 1. When you first run a good source code analysis tool on a code base that has
been around for some time, there will be lots of results to sift through. The best practice here is to separate the backlog of existing issues from new ones being created from that point forward. There are a few ways to do that, which could be its own deployment best practices paper, but the key thing is to get strict about new issues being introduced (zero tolerance would be a good place to start) and either gradually introduce backlog bugs to developers who will clean them up during implementation, or explicitly allocate time for your development team to aggressively burn down the backlog in a shorter period of time.
3. Measure Your Progress
This is true for any new business or engineering process, and source code analysis is no exception. Most decent source code analysis tools include a reporting interface that will allow you to automatically collect a set of core metrics. Use that data, share it with your engineering management (and the whole team if that fits your development culture) and get disciplined about monitoring these key performance indicators (KPIs). This is the exact approach Polycom took:
One of the disciplines that was key to the successful rollout is the use of Klocwork reports to give Polycom management greater visibility. “We report on Klocwork every week at our operations review. It’s in front of everyone’s eyes, and the managers know how many defects they have in their queue, how many are being fixed, and what the trend lines look like.”
Here are three must-have metrics you’ll want to collect on Day 1, and on a regular basis thereafter:
• Distribution of defect types across code base. This is a “state of your code base” view to see what you’re
dealing with and will help you immediately prioritize clean-up based on defect types that are important to you.
• Defect injection rate. Once you know the current state of your code, you’ll want to know if it’s getting better
or worse, and that’s the defect inject rate. Track this every build since it will show you which way the problem is trending, and also allows you to gauge the effectiveness of your deployment.
• Cost of fix activity. This can be discovered by identifying where bugs are being fixed — during implementation
(pre-check-in) or after the integration build. The metric here is really the inverse of defect injection since it’s identifying where defects have been removed rather than injected, and it gives you the opportunity to start fine tuning your deployment. You’ll want to know whether bugs are being fixed during implementation (where they are cheapest to fix) or later at build time. As long as they’re getting fixed before testing and not leaking later into
Systematically repeat this process for every team, on-board them the same way, share lessons learned, and be consistent.
HOW TO BUILD A BUSINESS CASE AND SHOW ROI
The basis for a source code analysis ROI model goes back to the very first chart shown in this paper — the more bugs you find during implementation, the more money you’ll save. As mentioned, everyone understands this, but the trick is to develop a model that’s conservative, grounded in some real numbers that make sense, and will get buy-in from others in your organization.A slightly modified version of that chart to assess the impact of source code analysis would look like the one shown below.
Figure 2 | Defect Detection vs. Person Hours to Repair
Explanation of the Assumptions Used
• Found @ Implementation = 15 minutes. Before code check-in is the optimal place to find and remove bugs,
and when you think about what’s required to fix a bug at the desktop, it’s a fairly conservative number. A developer is in-context working on a particular file and the source code analysis tool automatically finds a list of potential issues in that local file, complete with traceback information that steps the developer through the code. For most bugs found here, the fix is very quick.
• Found @ Integration Build = 1 hour. When bugs are identified, then triaged at integration build, it requires
developers to understand, interpret the bug report, check out the file in question, take some time to get back into the context of that code, make the fix, resubmit, and wait for the subsequent build to make sure it has been fixed. Also, there’s an inevitable administrative overlay to this model, which adds to the person hours, since it requires triaging of bugs from a central repository and distributing them to the developers.
• Found @ Test = 12.5 hours. Now you have test engineers involved who need to file a bug report in their bug
tracking system, which eventually gets to the developer, who then needs to go through a traditional debug cycle, isolate the issue, fix and resubmit code, followed by a subsequent re-test.
• Found @ Production = 25 hours. This number can vary widely depending on the type of software being
developed, but there are significantly higher levels of effort required to fix these issues when you consider the involvement of support teams, account management teams, not to mention the cost of brand damage, etc.
Applying These Assumptions
The simplest way to apply these assumptions is to just assign an average savings per defect found by source code analysis, which will result in a straightforward cost avoidance ROI. That’s what Johns Hopkins/DARPA did, using an estimated savings of 4 person hours per bug:
“We had a very tight schedule and without Klocwork, we would have had difficulty meeting our objectives on time. Even with a relatively small development team, we estimate 900 person hours saved by using Klocwork.”
Alternatively, you can apply the same principle using cost of defects (rather than person hours), and come up with a straightforward, conservative ROI. Spirent took this approach and using a conservative estimate for defects found per release, came up with a compelling business case:
“We’ve calculated that the cost of fixing a defect in the field is $962 and the cost to fix it with static analysis is $96. With a large code base such as ours, we could easily find hundreds of bugs in one release cycle, so the ROI is very compelling and easy to justify.”
Another approach which is more involved, and is best used when you have more detailed source code analysis usage data to work with, involves calculating person hour savings based on a distribution of where the defects are actually found. After some usage with the tool, you’ll have data on fix activity at both the implementation and integration build, as well as data on what bugs weren’t fixed at all, which represents bug leakage into testing or beyond (hopefully there are none of these!).
Let’s assume a modestly sized project of 750,000 LOC and a 1 bug/KLOC rate, which would translate into 750 bugs that need to be repaired. Your usage data shows that 90% of SCA-found bugs are getting fixed at the desktop, 10% at integration build, and your zero-tolerance program for SCA bugs is ensuring none are leaking into testing or beyond. You then need to paint a picture of life without source code analysis, which is a bit trickier and requires making some assumptions. For this example, we’ll use a conservative assumption that 50% of the bugs found by the tool (remember, these are real bugs that exist in your code!) would be found within that same time period, either at testing or in the field.
Person hour Effort % Defects Repaired w/SCA % of Defects Repaired w/o SCA
Developer Desktop .025 90% –
Integration Build 1 10% –
Testing 12.5 – 25%
Field 25 – 25%
Savings Summary Total Person Hours Spent 244 person hours 7,032 person hours Total Time Saved w/SCA 6789 person hours / 3.4 person years saved
Notes:
1. These numbers represent bugs found by source code analysis and not all bugs that could exist in your software. 2. The “Without SCA” scenario assumes only 50% of the bugs will be discovered during the time period of the ROI.
Rogue Wave provides software development tools for mission-critical applications. Our trusted solutions address the growing complexity of building great software and accelerates the value gained from code across the enterprise. The Rogue Wave portfolio of complementary, cross-platform tools helps developers quickly build applications for strategic software initiatives. With Rogue Wave, customers improve software quality and ensure code integrity, while shortening development cycle times.
Using these conservative assumptions, you can see that even a modestly sized code base can yield significant productivity gains across your team. There are, of course, some nuances to this that will be unique to every deployment (mix of backlog defects vs. newly injected ones, different cost estimates, etc) but the basic premise is that you have a universe of bugs that must be repaired within a period of time, and the cost of repairing them can be significantly less if you address them in a more cost-effective manner.
Once you’ve made these calculations, you can convert person hour savings into financial costs using your own loaded labour rates, apply any additional conservative assumptions as you see fit, and work with this as a baseline model for cost justifying your investment in source code analysis.
CONCLUSION
There is much evidence that source code analysis can find a large number of significant issues in your code that need to be addressed. Furthermore, there’s no other automated code analysis technology that can help you find these issues as efficiently as source code analysis. The key to cost justifying the use of SCA technology on an ongoing basis within your organization is to combine real, tangible source code analysis deployment data with some conservative cost-savings assumptions and come up with a model that’s accepted within your organization and that you can use throughout the life of your deployment.
