Accelerate Development Velocity
and Reduce Costs with Automated
Code Testing
The New Business and Development Imperative
for Financial Services Applications
Technology Innovation Brings New Challenges to Financial Services
Economic, regulatory, competitive and technological changes are reshaping the financial services information technology landscape. Many financial services IT departments, once charged with back-office development and maintenance of core systems, now find themselves thrust front and center – tasked with the role to develop new products and services to address evolving business trends such as:
• Customer mobility and higher expectations for accessibility to services • Stronger regulatory requirements and more sophisticated hacker threats • Business agility and the ability to deliver services on-demand and in real-time
Increase in Technological Complexity
One of the biggest challenges for development teams that create financial systems, such as banking applications, trading systems, and credit card processing modules, is dealing with the complexity of these applications and integrating these new services with core or legacy applications and systems.
As customers become more connected, more mobile and less captive – they expect the ability to access services over a variety of channels. The World Payments Report 2011, conducted by RBS examining the latest developments in the global payments landscape, found that electronic and mobile payments accounted for an estimated 22.5 billion transactions in 2010. If growth continues at the same rate, the study predicts mobile payments will represent 15% of all card transactions by 2013, and will exceed credit and debit card volumes within 10 years. According to NACHA, the electronic payments association responsible for governance of the ACH network - these statistics reflect an ongoing trend of preferences for fully electronic payments. “Consumers and businesses alike are looking for expediency and convenience in payments, with a stroke of the keyboard or the touch of an app to make a one-time payment or to enroll in recurring payments,” said Janet O. Estep, NACHA president and CEO. As a result, financial companies looking to remain competitive not only need to provide new online billing and transaction services but also make these available on emerging mobile platforms such as the iPhone and Android. Another major trend impacting IT is the proliferation of regulations – SOX, PCI, GLBA, CA SB 1386 and BASEL III…to name a few, fueled by the global credit crisis and increasingly pervasive cyber threats. The financial services industry has always been a heavily regulated one, but recently, these regulations have been coupled with harsher penalties for breach of compliance.
All of these mandates require financial services organizations to perform a higher level of due diligence to ensure the confidentiality, integrity and availability of customer data and transactions. Software applications are complex, and hidden vulnerabilities can introduce risk. Because financial institutions often cannot be sure that their applications are secure, they must develop and implement costly and inefficient compensating controls.
A third major trend is the increasing dependency on software applications to run business and deliver competitive advantage, from online banking to claims servicing to trading systems. From the single desktop of a hedge fund startup to Goldman Sachs, computer code is now integral to a lot of the trading activity on Wall Street. Computer-aided high frequency trading is estimated to account for 70 percent of total trade volume. These complex algorithms are built to respond instantly to rapidly shifting market conditions, taking into account thousands to millions of data points every second. And due to the complexity of these algorithms, they can create unexpected behaviors— overwhelming the systems they were built to analyze. On May 6, 2010, the Dow Jones Industrial Average inexplicably experienced a series of drops that came to be known as the flash crash, at one point shedding some 573 points in five minutes. Observers attributed this sudden drop, and subsequent ones since then, to these powerful and super-fast trading algorithms. While these events may have been triggered by misuse, they do illustrate the inherent risks associated with the malfunction of increasingly complex algorithms and applications. This rise in complexity of applications across the industry is reflected in the increasing size and complexity of their codebases. A Gartner study conducted in early 2000 estimated that the world’s active business applications accounted for close to 300 billion lines of codes, with an expected growth of 5 billion lines of code per year. Manually testing this volume of code is neither practical nor possible.
The Business Impact of a Software Defect
Traditional approaches to testing begin in QA with automated and manual functional and performance testing to ensure the application functions and scales as intended. But what about the unintended issues—the ones that cannot be tested using traditional approaches? In some cases, development velocity and the rate of change is so rapid that there isn’t even time for a full QA process, such as the case with high frequency trading applications. The cost of a defect could mean an inaccurate calculation or latency of trades, which could translate to millions in lost revenue or penalties – such as in the case of the flash crash.
According to the National Institute of Standards and Technology (NIST), the cost benefits of finding and addressing defects early are staggering. For every dollar spent on addressing defects during the coding phase of development, it will cost an organization thirty dollars to address if detected in production.
According to a 2011 commissioned study conducted by Forrester Consulting on behalf of Coverity, respondents from financial services institutions stated that the most serious consequences of finding defects late in the software development lifecycle are:
• Increased costs (77% of respondents)
• Delays in time to market/project release schedule (35%) • Hurt developer productivity (28%)
• Accumulation of technical debt (19%)
And that is if these defects are found before production.
System downtime of online banking applications have been the cause of front page headlines for more than one banking institution in the recent past, resulting in millions of lost revenues, lost customer satisfaction, and brand damage. Some of the worst software failures of 2011 include:
• When hackers stole bank account details for 200,000 Bank of America customers by exploiting a garden-variety security hole in the company’s website, the company’s stock plunged 15 percent. This also impacted the stocks of rival banks.
• A US based financial services organization revealed that a vulnerability in its portfolio information system resulted in more than 1 million individual records to be inappropriately accessed.
• In February 2011, the SEC charged three AXA Rosenberg entities with securities fraud for concealing an error in the source code of the quantitative investment model used to manage client assets. This error from a software defect resulted in $217 million in investor losses plus SEC fines totaling more than $240 million to the firm.
• Computer system problems at Mizuho, one of Japan’s largest banks, resulted in a nationwide ATM network of more than 5600 machines going offline for 24 hours. Internet banking services were shut down for three days and resulted in delays of salary payments worth $1.5 billion into the accounts of 620,000 people and a backlog of more than a million unprocessed payments worth around $9 billion. • A technical glitch in Australian Commonwealth Bank’s computer system enabled people to withdraw any amount of cash up to their
daily limit, regardless of how much money they had in the bank. More than 40 ATM cash machines across Sydney and Melbourne were affected by the problem. The technical glitch reportedly lasted for about five and a half hours and caused a stampede on ATM machines as people attempted to exploit the “windfall.”
It is clear that even large, multinational companies are struggling in their efforts to secure themselves against exposure of sensitive data and assets from system malfunctions and hacker exploits of defects in their applications.
Increased Pressure on both Developers and QA Testers
This emphasis on faster delivery of new complex and secure applications adds pressure and workload on both developers and QA testers even as IT budgets and resources remain flat. In the Financial Services World Quality Report co-published by Capgemini Group and HP, 80 percent of respondents indicated their company is continuing to invest in new application development efforts but half of the survey respondents indicated that resource cuts and heavier project loads have forced their project teams to take on more work. This increased workload on QA shortens the time available for testing and is therefore pushing testing upstream earlier in the application lifecycle.
Traditional Testing and Development on Demand – A New Approach is Needed
The challenges facing IT and development teams in financial services are not unlike those in other industries such as manufacturing, telecom or healthcare. Regardless of industry, there is a constant need to build new functionality while maintaining legacy applications and to get products released as quickly to market as possible. These increased pressures mandate the need for greater efficiency, more consistent QA methodology, better automated testing effort, and the need to expand testing beyond its traditional definition and move it upstream into development.
The Benefits of Development Testing
Development testing is a set of processes and technologies, such as static analysis, to help development organizations find and fix software problems early in development, as the code is being written, when it is the fastest and most cost effective to address.
Development testing augments traditional testing, including QA functional and performance testing and security audits, providing development teams with a quick and easy way to test their code defects in a non-intrusive manner so development stays focused on innovation while management gets visibility into problems early in the cycle to make better decisions.
Development testing helps to achieve:
• Application resiliency and security – by automating code testing during development, defects can be fixed earlier in the development cycle. Quality and security defects which can be hard to identify, hard to reproduce or happen only infrequently can be easily identified and fixed as the code is written, before unit testing and before the code even reaches the QA and security audit teams.
• Increased development velocity to meet rapid changes – automating code testing can trim down the time needed to review code manually or through other testing methods, empower the development team to meet business requirements to release products and services to market faster.
• Better development and QA collaboration – by integrating development testing into an established ALM platform, developers and QA testers gain a shared common workflow and view into development defects and code quality issues right alongside functional and performance defects. It also allows developers to establish code quality and security KPIs, and only release to the QA team for testing when the code quality is within acceptable levels. On the flipside, QA teams gain visibility into development related defects and can plan their testing strategies appropriately.
• Better development and security audit collaboration – by bringing security into the development phase of the software development lifecycle, development testing puts the responsibility of easily overlooked, code defects that could lead to vulnerabilities into the hands of the development team, allowing the security audit team to focus on more complex and critical application vulnerability testing. • Policy management and compliance – a common workflow for code testing promotes consistent policies and controls across internally
According to a 2011 commissioned study conducted by Forrester Consulting on behalf of Coverity, respondents from the financial services industry stated that development testing is more important today than it was two years ago due to:
• Increased awareness of the time savings of finding and fixing defects early in the lifecycle (67% of respondents) • Increased pressure on the development group to find and fix defects (53%)
• Increased awareness of the cost savings of finding and fixing defects during development (36%)
• Increased awareness that not all code defects can be found in traditional functional or performance testing processes performed by a dedicated QA team (31%)
Coverity Development Testing Platform - Designed for Developers, by Developers
Coverity development testing helps financial services institutions effectively manage the quality, security, and complexity of code—and the efficiency of the teams that develop it. By setting standard software development policies, based on the business priorities, automatically testing code in development against those policies, and controlling internal teams, outsourced teams, projects, and third-party suppliers against common and defined metrics - development organizations gain visibility and early warning of risks across the software supply chain.
Defining Software Policies and Thresholds
Coverity® Integrity Control lets you centrally define software development policies based upon business priorities. Once defined,
the policies are centrally published and then shared with geographically dispersed teams and with software suppliers. Specific policies which can be defined include:
Quality policies: You can set policies for defect density, critical defects and uninspected defects. Defect density represents the
number of outstanding high or medium-risk defects per 1000 lines of code. Critical defect policies can be established as well as thresholds for uninspected defects since these defects could represent a risk to the overall code quality.
Security policies: You can establish and enforce policies for defects identified as security risks by the industry standard
Common Weakness Enumeration (CWE), and establish policies for security defect density and web security defects.
R&D productivity: You can establish policies for your internal teams and third-party suppliers for critical metrics tied to R&D
efficiency. Overly complex code is a leading contributor to technical debt and can lead to maintenance issues and difficulty with future innovations. Comments are also particularly important when taking delivery of code from a third-party or if you have a large development department with high attrition rates. You can manage technical debt inherited from legacy systems or suppliers by establishing thresholds for acceptable comment density levels and cyclomatic complexity which measures the number of linearly independent paths through a program’s source code.
Usage and savings policies: You can establish policies around the usage of Coverity® Static Analysis and Coverity® Dynamic
Analysis. Policies can be established for the number of active users, projects and lines of code being scanned. This can be critical in enforcing code quality and security across your organization and supply chain.
Testing Often and Early
Coverity development testing solutions enable you to test against established policies while the code is still in development, and where issues are least expensive and time consuming to fix. Coverity Static Analysis and Coverity Dynamic Analysis use sophisticated algorithms to identify and triage high risk defects that could result in software crashes, security breaches, or safety issues. It enables you to find hard-to-spot issues such as null reference pointers, memory leaks, and potentially exploitable security flaws in the largest, most complex codebases. In addition, Coverity ships with a highly tuned version of FindBugs™, a popular open source tool for finding defects in Java
codebases, with no extra installation required. This allows your team to manage FindBugs™ identified defects alongside Coverity identified
defects from a centralized defect management console. Once defects are found, your developers are automatically notified of defects within their existing workflow, prioritized by risk and impact, so they know which problems to fix first. They have access to a rich defect knowledge base, along with source code navigation to show them exactly where the defect exists in the code and guidance on how to fix it. Because many organizations leverage shared code across projects and services, Coverity will also show the development team all of the places across the shared code where that defect exists, and will apply the fix in all places. Because Coverity solutions are designed with developers in mind, Coverity’s testing platform fits within existing development workflows, enabling developers to quickly identify quality, security and safety defects from within their IDE at their desktop, or as part of the continuous or central build system.
Controlling Risk across Development Teams, Projects, and Suppliers
Once you have established your policies and tested against them, it is critical to have visibility into the risk across the organization and the software supply chain. Coverity Integrity Control provides you with a visual representation of the areas of risk across your projects and teams. As a Development Manager or Application Owner, you can view a hierarchical heat map that is tailored specifically to the needs of your organization. You can track distributed teams to ensure they are executing consistently and will be able to quickly address any potential areas of risk and skills gaps. It is able possible to track by product portfolio or by project component delivered from each team. Coverity Integrity Control provides the visibility and control needed to consistently measure internal teams as well as suppliers against the same standards for quality and security – with the ability to audit SLA violations on-demand. You can drill down into each policy to pinpoint the full context of the code problem, identify the specific policy in violation and where it originated. An updated risk profile is produced with every code iteration and test.
In addition, you can easily notify teams and third-party code suppliers of code governance violations by sending them a Coverity Software Integrity Report summarizing the high risk defects that exist in their software, or violations from established policies. Once developers and third-party suppliers receive the automatic notifications, they can quickly begin the triage or inspection process to fix new defects. This can also be used to help with internal audits as part of the compliance process. For example, the CIO of a multinational stock trading company uses Coverity to help them verify code integrity as part of their SEC audits.
Integrated into the Software Development Lifecycle
Coverity enables you to choose best of breed technology while fitting into your existing software development lifecycle process such as Agile. Seamless integration with IDEs like Eclipse or Visual Studio lets you view defects and understand their severity and impact. Once the code has been analyzed, you can check it into the continuous integration server or central build system where the analysis engine can evaluate the cumulative changes of the entire development team. To save time, you can also choose to utilize incremental analysis which only analyzes the files which have been changed or those affected by the change instead of the entire code stream. By scanning code from the desktop, you are able to address security and quality issues immediately – as part of your development process.
Developers can test their code within the Eclipse or Visual Studio IDE on their desktop
One of the most common practices of Agile development is continuous integration (CI). By increasing the frequency of integration that CI provides, delivery teams improve their visibility of the overall quality of the software. Integration issues, build problems and code conflicts are surfaced more quickly allowing faster remediation. In order for a development testing solution to work in an Agile environment, it is essential that the analysis is done as frequently as the source integration happens. The analysis needs to be automated, fast and scalable especially when the development team is large. Coverity Static Analysis is integrated with build tools such as Jenkins, which enables an automated continuous process for code assurance.
Fostering Development and QA Collaboration
Through the out-of-the-box integration with HP Application Lifecycle Management (ALM), Coverity development testing results are automatically surfaced in the ALM and HP Quality Center workflow, providing development and QA with a single platform and common workflow for collaboration through visibility into defects identified in development. With every code change, Coverity automatically tests the code for defects, surfaces them in HP ALM, and links them to the corresponding business requirement so development and QA know where to focus their efforts, reducing risk of releasing defects into production without impacting time, cost or speed of deployment. This level of collaboration, defect traceability and visibility within the existing workflow is critical to agile organizations trying to rapidly ship products to market while maintaining acceptable levels of quality.
Summary
Developers are acutely feeling the pressure of delivering services faster, and the business cannot tolerate the customer satisfaction loss and brand damage associated with today’s headline making software failures. It is no longer sufficient to begin testing in QA, or for development to be informed of defects which need to be addressed after the QA and security audit process is complete. Development organizations need an automated solution for assuring the quality and security of their software that keeps up with the rapid iterative development process.
However, technology adoption in development will be limited unless it is effortless for the developer. If testing results are not presented to the developer in a way that is actionable, relevant, and integrated into their workflow; the troubleshooting and fixing of defects will slow down the entire process. Given the time to market pressure on development, defects will likely go unresolved, or go undetected altogether. Coverity provides the industry’s first developer-friendly and enterprise-ready development testing platform, empowering development organizations to adopt development testing as a seamless part of the development process.
Experience that Matters
Coverity works with 25 of the world’s leading retail and investment banks, securities firms and stock exchanges to help them meet the highest levels of application quality and security by integrating development testing into their software development lifecycle. Coverity serves as the development testing gate for a leading stock exchange which trades approximately 1.46 billion shares each day. A top U.S. bank with approximately 16 million unique website visitors per month estimates that Coverity helped them reduce the amount of time required to find and fix defects by 99%. They were able to save 495,000 man hours in one project alone, thus accelerating their time to market. Coverity is used by a leading UK bank to test their finance software used in over 3,000 of their branch offices for proposing new products to 16 million customers and small businesses. A leading provider of mutual funds and trading platform technology uses Coverity to regularly test over 5 million lines of code that is delivered by a team of 80 developers.
For More Information
To find out how Coverity can help your organization improve the quality and security of your software and how it can be integrated into your software development lifecycle, contact your Coverity representative or visit us at www.coverity.com.
