Solving the Software Quality Challenges of Agile Development

Solving the Software Quality Challenges

of Agile Development

Solving the Software Quality Risks of Agile Development

Agile software development is a series of iterative and incremental development methods in which requirements and solutions evolve through team collaboration. The software industry has embraced Agile and it is estimated that over half of software development projects use Agile methods.

With all of its benefits, Agile is not a silver bullet. And when it comes to maintaining or improving software quality, there are still some serious challenges. As organizations seek more rapid software development cycles, new tools are emerging within continuous integration environments that can streamline the process of development but often make finding and fixing defects more difficult.

Agile’s Software Quality Challenge

The underlying cause of software quality issues lies in the conflict between modern software development, with its shorter release cycles, proliferation of components, and numerous product variants; and the attempt to maintain code quality using traditional tools and processes. In particular, with an Agile approach, a lot more artifacts (components, variants, releases etc.) are generated during a short period of time. As a result, the task of making sure these artifacts are bug free is particularly challenging.

This has its root in two practices:

> Component and software reuse. This is a good practice in Agile software

development, often leading to significantly lower development costs. However, it creates a large number of variants. For example, software produced for the telecom, mobile, aerospace, automotive, medical device, and consumer electronics industries, contains significant quantities of product variants that share software components.

> Parallel branching. This is the predominant strategy employed to streamline software development and support release management needs. For example, it is very common for software companies to maintain many old releases in the field even while working on a new release. In some organizations, it is also necessary to maintain multiple customized versions, each specialized for a specific customer. Developers continuously make code changes, including bug fixes, in these versions. Keeping these versions in sync is an enormous challenge.

The upshot of this: Code containing bugs is reused. Copy-paste is a widespread

programming practice used to save time. Research shows that software typically contains 15-50% duplicated code. In many cases, an entire component is reused and modified, even within the same branch. When a component of code is reused, buggy code within it is also propagated. Currently there is no effective method to keep track of these duplicated bugs.

SCM Systems Are Necessary Tools, But Wrong for Tracking Bug Fixes Software configuration management (SCM) systems are designed to track and control changes in source code and are an important component in Agile development. Software companies make well-intentioned attempts to ensure bug fixes are properly ported into releases. Often, the SCM solution is a key component in that attempt.

However, the problem of ensuring bug fixes are properly ported is much bigger than many

teams think. Companies spend hundreds of thousands of dollars to buy static analysis tools

to find unknown bugs in large codebases. It costs even more to fix the bugs found in QA,

development and production. But, in spite of all that effort, companies consistently fail to

eliminate those bugs from releases.

Why? Changesets in development branches often end up missing in production releases.

Given the devastating consequences of failing to eliminate security or other important types of bugs (recalls, hot fixes, etc.), it is useful to identify the common causes of this problem.

1. SCM Limitations

> Based on the history in the SCM log, companies build in-house tools to track the target branches where a bug fix has been applied. However, SCM systems rely on manual input. Developers must take the “right” action during merge or key in the “right”

information. But developers don’t always use the integration command to port bug fixes. Instead, the bugs are manually fixed in other branches. And that leaves no trace in the SCM log.

> SCM systems don’t understand code and treat it as text. For example, SCM systems cannot track duplicated content. And if a file is removed and added back in, SCM systems cannot track the changes between these two files. Additionally, if a change has been made to one branch, even if the change is superficial (e.g. a variable rename), the SCM will fail to perform the integration and the developer will have to decide “manually”

what action to take.

> Legacy SCM systems cannot perform even basic changeset tracking. Regularly used in development shops (CVS, Borland StarTeam, and Rational ClearCase Base) these earlier-generation SCM systems don’t support the notion of a changeset (or changelist).

They were designed to do one thing, track revisions in files. Naturally, they fail in their attempts to track changesets. This means the developer is unable to determine what the complete change is that needs to be applied—especially if that change spans multiple segments of code in multiple files. This makes tracking a bug fix nearly impossible.

Manual workaround “rodeos” become the methodology teams saddled with these older systems are forced to use.

2. Regular-Expression-Based Tool Failure

Another method often attempted is keyword search or regular expression-based tools.

These are really lacking in their scope of usefulness as well. Here’s why:

> Regular expressions to capture a bug across multiple segments of code are virtually impossible to write. It is extremely difficult to determine the right query to get the results you want. A bug tends to involve many snippets in different files. A simple keyword search or regex-based search cannot capture this complexity.

> The results are usually very noisy, since the search lacks context. Search results contain extraordinary volumes of irrelevant results. The overhead of processing them and narrowing down the candidate code snippets is prohibitive.

> It is impossible to be certain that the results are complete. For instance, cases including examples where buggy code has been copy-pasted into another area, and then

modified, are consistently missed.

3. SCM System Migration Increases Defect Issues

Agile development has become very popular, and as a result, many modern SCM systems

are designed around this methodology. Git’s fast branching and merging, for example,

makes it an appealing SCM for Agile environments. Many companies are now moving

towards more Agile-friendly SCM systems. So it’s common for a company to be using

one SCM system, such as CVS, for an old product, while migrating to GIT or Perforce for

a newer product line. These migrations tend to be a lengthy process, and development teams are forced to manually inspect all code to assure every change made in one system is also made in the other—there is no cross-SCM checking mechanism available. The opportunity for defects to make it into a production release increases significantly in this situation.

What Is the Solution?

Any robust solution to the problem of ensuring bug fixes are properly ported into releases must:

> Parse the code: Unless a tool can parse and understand the code, it’s impossible to identify copy-pasted segments or files, tolerate superficial changes and make structural and semantic matches.

> Not rely on manual input: If a human is involved and the process is not fully

automated, it’s impossible to avoid operator mistakes. This is the reason we are looking for tools to ensure that bug fixes are in the release in the first place.

> Track replication: Based on our studies of open source and commercial code, 20-40%

of the code within a single code base is replicated (for the Linux kernel, for example, the percentage is in the low 20s). Furthermore, about 2/3 of these replicated code segments are modified. If a tool cannot account for 20-40% of a codebase it’s useless.

> Be automated: The only way for a solution to scale is to be fully automatable. I fixed a bug, now give me a report, or, take all the bugs from a development branch and tell me if they are in the release. Otherwise, there is no way that there is going to be wide adoption.

> Identify complete bugs (not just single snippets of code): The vast majority of bugs in mature codebases involve multiple snippets of code and usually multiple files.

If a solution does not take the complete set of changes into consideration or fails to consider the context it will overwhelm users with results that are mostly noise.

Pattern Insight’s Code Insight

Code Insight from Pattern Insight is the only software solution that ensures every instance of a bug is found and never released again. It is based on patent-pending fuzzy matching technology, which can tolerate any variable name change or statement insertion and deletion.

Pattern Insight’s Code Insight has the characteristics required to provide automated defect elimination:

> Fast: Returns results in seconds, even for billions of lines of code

> Accurate: Extremely low false positives

> Easy-to-use: Fully integrates with all SCM systems and is capable of being used in any development, build, and release process

Additionally, it:

> Finds all similar code patterns within a single version or a branch of a code base.

> Compares two versions or branches of a single codebase, and identifies all code segments that are similar or identical between them.

> Compares two different code bases and identifies all code segments that are similar

Code Insight can be used in numerous workflow scenarios:

Ensure releases are clean. For Build/Release Owners, Pattern Insight’s Code Insight easily integrates into the release process or continuous integration to identify previously fixed bugs in releases going out the door. For example, one of our customers in the mobile device industry built a catalog of thousands of security and other Priority-1 bugs and runs nightly reports to determine if any of these have “leaked” into its 100s of builds. If a match is found, the build is blocked and the developer is notified automatically.

Figure 1: Ensure releases are clean.

Eliminate all instances of bugs in development. Developers want to ensure that a bug has been fixed across all locations branches and components in the development process.

In this “catch point” use, the developer uses Pattern Insight’s Code Insight to quickly run a report to find where every instance of a bug is and fixes them immediately. More automatically, the developer can be informed in code review or code check-in.

Figure 2: Eliminate all instances of bugs in development.

Ensure that all changes are ported into the final release. Release engineers deploy

Pattern Insight for use beyond bug fixes. Code Insight can be used to automatically ensure

that changes made in separate branches make it into the final release. The number of the

changes checked can be hundreds or thousands in one single run. This process takes a

matter of minutes, whereas manual verification may take months.

Figure 3: Ensure that all changes are ported into final release.

Pattern Insight matches other workflow scenarios as well:

> Ensuring that all changes are migrated properly when the underlying platform is upgraded (for example, from Android 2.0 to Android 3.0).

> Ensuring that patches released by open source software components or third-party commercial software components are incorporated correctly.

> Ensuring that a code style deemed to be detrimental to software quality is not used.

And in the case of SCM system migration, by looking into the code itself with Code Insight, we provide a means to solve this problem that avoids manual inspection. Here is a sample workflow:

> Code Insight deploys two SCM connectors to CVS and GIT

> The connector knows how to pull code and changes from its corresponding SCM system

> Once the code and changes are pulled into the system, Code Insight performs the analysis based purely on the code itself (this analysis is SCM agnostic).

> The resulting report is presented using SCM-specific terms, such as commits in GIT or a set of file revisions in CVS.

Conclusion: To Maintain Code Quality in Agile Environments, You Must Gain Insight Into the Source Code Itself

Pattern Insight is the only means to pinpoint known defects and missing changesets in your source code. It is essential to gain insight into the source code itself versus relying on metadata. You need an accurate assessment versus “best guesses” as to the quality of your software. The manual processes that engineering departments rely on are risky and inadequate in light of the explosive growth in artifacts generated in the shorter development cycles. And Agile development becomes more popular, newer SCM systems, designed around Agile methodology present additional challenges. These factors have caused major software companies, including Cisco, Qualcomm, Motorola, Wind River and EMC, to adopt Pattern Insight to make sure they never release software with bugs they’ve already fixed.

To learn more about Pattern Insight’s solutions, go to PatternInsight.com.

Copyright © 2012 Pattern Insight Inc. All Rights Reserved.

Pattern Insight Headquarters 465 Fairchild Drive, Suite 209 Mountain View, CA 94043 P: 866 582 2655

F: 408 573 7855

E: info@patterninsight.com

PatternInsight.com