The Forrester Wave : Application Security, Q4 2014

(1)

Security, Q4 2014

by Tyler Shields, December 23,2014

Key TaKeaways

HP, IBM, Veracode, whiteHat, Contrast security, Quotium, and Checkmarx Lead

Forrester analyzed the application security testing market (static analysis, dynamic analysis, and instrumented/interactive technologies). The results of the analysis find that HP Fortify, IBM, Veracode, WhiteHat Security, Contrast Security, Quotium, and Checkmarx lead the field. Beyond Security, Coverity, Qualys, and Virtual Forge offer competitive options, while Trend Micro lags behind.

s&R Pros Look For solutions That emphasize accuracy, Integration, and scalability

The application security testing market is steadily growing because S&R pros increasingly trust application security testing providers to act as strategic partners, advising them on top app security decisions. Solutions with the highest accuracy of results, best integration points, and the capability to grow and scale will continue to find success.

Combining assessment Methods Provides Differentiation In The Market

By combining static, dynamic, and instrumented assessment technologies, vendors are creating a platform-based application security assessment model. Each individual assessment technology contains delivery weaknesses when compared with the others.

access The Forrester wave Model For Deeper Insight

Use the detailed Forrester Wave model to view every piece of data used to score

participating vendors and create a custom vendor shortlist. Access the report online and download the Excel tool using the link in the right-hand column under “Tools & Templates.” Alter Forrester’s weightings to tailor the Forrester Wave model to your specifications.

(2)

wHy ReaD THIs RePoRT

In Forrester’s 82-criteria evaluation of application security vendors, we identified the 12 most significant service providers in the category — Beyond Security, Checkmarx, Contrast Security, Coverity, HP Fortify, IBM, Qualys, Quotium, Trend Micro, Veracode, Virtual Forge, and WhiteHat Security — and researched, analyzed, and scored them. This report details our findings about how well each vendor fulfills our criteria and where they stand in relation to each other to help security and risk professionals select the right partner for their application security requirements.

notes & resources

Forrester conducted product evaluations in april 2014 and interviewed 12 vendor and user companies: Beyond Security, checkmarx, contrast Security, coverity, HP Fortify, iBM, Qualys, Quotium, trend Micro, Veracode, Virtual Forge, and WhiteHat Security.

2014

tools and technology: the Security architecture and operations

Playbook

by tyler Shields

with Stephanie Balaouras and Jennie Duong

2

2 5 7

(3)

s&R PRos MUsT BUILD seCURITy InTo THe aPPLICaTIon LayeR

Application security remains a crucial component in keeping enterprise servers and data secure. Many firms have rushed to bring applications online, building out consumer-facing websites, buying commercial off-the-shelf (COTS) products, and developing mobile applications to enable and engage with their customers and partners without thinking about the security of the application itself. As a consequence, businesses are exposing their most sensitive corporate and customer data to possible external threats and breaches.

selection Criteria Target Functional security Capabilities and strategy

S&R pros are looking to work with vendors to detect vulnerabilities more efficiently and effectively, to avoid exposing business-critical security issues, as well as to improve overall business and development efficiency. The vendors assessed meet the growing demand across all businesses without compromising the security needs of their clients’ operations. After examining the current trends in the market, user needs assessments, and vendor and expert interviews, we developed a comprehensive set of evaluation criteria for application security. We evaluated vendors against 82 criteria, grouped into three high-level categories:

■

Current offering. The vertical axis of the Forrester Wave graphic reflects the strength of each

vendor’s product offering, including its capabilities in general features (e.g., deployment model, scalability, targeted scanning), static application security testing features, dynamic applications security testing features, instrumented analysis features, reporting features and workflows (e.g., flaw descriptions, centralized policies, and workflow and remediation tracking), developer education and training, integrations (e.g., IDE integration, API access, patches, WAFs, and MDMs), remediation instructions, and customer references.

■

Strategy. The horizontal axis measures the viability and execution of each vendor’s strategy,

which includes planned enhancements, key technology partners, product focus, target market, cost, average sales prices, maintenance costs, and pricing structure.

■

Market presence. The size of each vendor’s bubble on the Forrester Wave graphic represents each

vendor’s presence in the application security market, based on its installed customer and product base, revenue, systems integrators, services, number of employees, and key technology partners.

VenDoRs oFFeR aPPLICaTIon assessMenT UsIng MULTIPLe TeCHnoLogIes

Forrester included 12 vendors in the assessment: Beyond Security, Checkmarx, Contrast Security, Coverity, HP Fortify, IBM, Qualys, Quotium, Trend Micro, Veracode, Virtual Forge, and WhiteHat Security. Each of these vendors has (see Figure 1):

(4)

■

Mindshare with Forrester’s clients. Vendors included are frequently mentioned in Forrester

client inquiries and other forms of client engagement relating to application security.

■

Ability to offer SAST, DAST, and/or IAST capabilities. The vendors evaluated offer

comprehensive approaches in static analysis (SAST), dynamic analysis (DAST), and instrumented/ interactive technologies (IAST) techniques in order to detect weaknesses and vulnerabilities in general code, web applications, mobile applications, and COTS product offerings.

■

Ability to provide easy deployment and integration for its customers. All vendors evaluated

could deploy either a scalable on-premises or cloud-based service to their customers. Vendors were focused on improving the security development life cycle for the enterprise buyer by creating a solution that integrated into existing models of analysis and third-party development tools.

■

Relevance to the application security market. Inclusion in this Forrester Wave means that the

vendor actively competes in the application security market, showing up in competitive use cases and discussions among experts and Forrester clients.

(5)

Figure 1 Evaluated Vendors: Product Information And Selection Criteria

Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

Product evaluated

Evaluated vendors Product version

Beyond Security AVDS 4.00 build 307

CxSuite 7.1.4

Contrast Enterprise* 3.1 Code Advisor 7.5 HP Applications Security Portfolio

(includes: HP Fortify Software Security Center, HP Fortify Static Code Analyzer, HP WebInspect, HP Fortify on Demand,

and HP Application Defender)* IBM Security AppScan Standard, Enterprise, and

Source

N/A

9.0.0

Qualys Web Application Scanning 3.2

Seeker 3.0

Deep Security for Web Apps* N/A Veracode Platform* N/A CodeProfiler 3.5 N/A Checkmarx Contrast Security Coverity HP Fortify IBM Qualys Quotium Trend Micro Veracode Virtual Forge WhiteHat Security

Vendor selection criteria

Mindshare with Forrester’s clients. Vendors included are frequently mentioned in Forrester client

inquiries and other forms of client engagement relating to application security.

Ability to offer SAST, DAST, and/or IAST capabilities. The vendors evaluated, offer comprehensive

approaches in SAST, DAST, or IAST techniques in order to detect weakness and vulnerabilities in general code, web applications, mobile applications, and COTS product offerings.

Ability to provide easy deployment and integration for its customers. All vendors evaluated could

deploy either a scalable on-premises or cloud-based service to their customers. Vendors were focused on improving the security development life cycle for the enterprise buyer by creating a solution that integrated into existing models of analysis and third party development tools.

Relevance to the application security market. Inclusion in this Forrester Wave™_{means that the}

vendor actively competes in the application security market, showing up in competitive use cases and discussions among experts and Forrester clients.

WhiteHat Sentinel WhiteHat Sentinel Source*

(6)

THe aPPLICaTIon seCURITy MaRKeT UnCoVeReD

The evaluation uncovered a market in which a platform approach strengthens the limitations of each assessment model, raising the importance of a multitechnology approach. Vendors that built a platform leveraging DAST, SAST, and even IAST saw higher marks than those vendors that focus on one or two technology sets (see Figure 2):

■

HP, IBM, Veracode, WhiteHat, Contrast Security, Quotium, and Checkmarx lead. While

each vendor designed and built its original solution from a different technology starting point, the end result for each of these vendors is a cross-technology solution. When sorted by technology cut, each vendor in this section excels at more than one solution, giving it the ability to increase the accuracy of its results.

■

Beyond Security, Coverity, Qualys, and Virtual Forge offer competitive options. Competitive

offerings from these vendors tended to focus on one specific area (DAST, SAST, or IAST) of the cross-platform solution. These vendors are working toward building an integrated platform but have yet to improve their nondominant offerings to the strength level that puts them in the upper echelon. These vendors should be chosen specifically if their dominant technology area is of a stronger concern to the buyer than the nondominant technologies.

■

Trend Micro is building up momentum from a late market entry. As a late market entry, Trend

Micro is lagging behind the other vendors in the space. Focused predominantly on DAST

assessments from the cloud, Trend Micro must expand beyond a single technology-based solution in order to remain relevant to the enterprise buyer. The level of resources available for continued innovation keeps Trend Micro in the picture if it can maintain its current pace of advancement. This evaluation of the application security market is intended to be a starting point only. We encourage clients to view detailed product evaluations and adapt criteria weightings to fit their individual needs through the Forrester Wave Excel-based vendor comparison tool.

(7)

Figure 2 Forrester Wave™: Application Security, Q4 2014

Risky

bets Contenders performersStrong Leaders

Strategy Weak Strong Current offering Weak Strong Go to Forrester.com to download the Forrester Wave tool for more detailed product evaluations, feature comparisons, and customizable rankings. HP Fortify IBM Veracode WhiteHat Security Contrast Security Quotium Checkmarx Beyond Security Coverity Qualys Virtual Forge Trend Micro Market presence

(8)

Figure 2 Forrester Wave™: Application Security, Q4 2014 (Cont.)

CURRENT OFFERING General features Static analysis features Dynamic analysis features Instrumented analysis features Reporting features and workflow Developer education and training Integrations Remediation instructions Customer references STRATEGY Product strategy Corporate strategy Cost MARKET PRESENCE Installed base

Revenue growth quarter over quarter

Revenue growth year over year Systems integrators Services Employees Technology partners Fo rr es te r’s we ightin g 50% 21% 22% 22% 5% 10% 5% 5% 10% 0% 50% 50% 50% 0% 0% 40% 0% 0% 15% 15% 15% 15% Whit eHat Securi ty 3.60 4.25 3.25 4.70 0.00 4.20 5.00 2.85 1.50 3.00 4.65 4.30 5.00 0.00 3.70 4.12 0.00 0.00 1.00 5.00 3.67 4.00 Vi rt ual Fo rg e 2.93 4.00 2.40 2.19 3.00 4.40 1.00 2.90 3.00 5.00 3.35 2.90 3.80 0.00 2.57 3.24 0.00 0.00 1.00 1.66 2.33 3.50 Tr end Micr o 1.82 3.70 0.00 2.38 0.00 3.10 0.50 0.75 1.50 5.00 3.30 3.60 3.00 0.00 3.12 3.68 0.00 0.00 1.00 3.35 3.65 3.00 Qual ys 2.49 3.65 0.00 3.98 0.00 4.20 0.00 2.55 3.00 5.00 3.95 4.30 3.60 0.00 4.31 4.78 0.00 0.00 5.00 2.34 3.65 5.00 Ve racode 3.92 3.80 4.00 4.38 3.50 4.80 5.00 4.50 1.50 5.00 4.65 4.30 5.00 0.00 4.26 3.90 0.00 0.00 5.00 4.34 3.67 5.00 Qu otiu m 3.21 4.40 2.80 2.59 5.00 4.60 0.00 4.90 1.50 5.00 4.65 4.30 5.00 0.00 1.96 2.58 0.00 0.00 0.00 1.00 2.66 2.50 HP Fo rt ify 4.42 4.68 4.80 4.48 5.00 5.00 5.00 4.90 1.50 5.00 4.65 4.30 5.00 0.00 4.15 4.56 0.00 0.00 1.00 5.00 5.00 4.50 Contrast Securi ty 3.97 4.25 4.50 3.82 5.00 4.00 5.00 3.85 1.50 5.00 4.00 3.00 5.00 0.00 1.37 2.36 0.00 0.00 0.00 1.00 1.34 0.50 IB M 4.62 4.55 4.60 4.56 3.50 4.80 5.00 4.85 5.00 5.00 4.35 3.70 5.00 0.00 4.60 4.76 0.00 0.00 3.00 5.00 5.00 5.00 Co ve rity 2.36 3.95 3.20 0.00 0.00 4.50 1.00 3.55 1.50 4.00 4.30 3.60 5.00 0.00 2.57 3.24 0.00 0.00 0.00 3.66 3.33 1.50 Ch eckmar x 3.12 4.25 3.55 0.89 3.00 4.80 5.00 4.45 1.50 5.00 4.30 3.60 5.00 0.00 3.34 3.24 0.00 0.00 3.00 2.32 3.32 5.00 Be yond Securi ty 2.46 4.55 0.00 3.68 0.00 3.60 1.25 2.35 1.50 3.00 4.30 3.60 5.00 0.00 4.50 5.00 0.00 0.00 5.00 3.68 2.99 5.00 All scores are based on a scale of 0 (weak) to 5 (strong).

VenDoR PRoFILes Leaders

■

HP Fortify offers an extensive application security solution. HP Fortify displayed strong

capabilities across the majority of our criteria in both static and dynamic analysis. Its product combines comprehensive static and dynamic testing and management across a multitude of languages and frameworks that allow customers to deploy and scale quickly. Customer

(9)

references were satisfied with the vendor’s ease of installation, ease of use, configurable scans, and administration; one customer reference was “very happy with the SDLC-based dynamic application security program built around HP WebInspect and would recommend it to any other organization, even those with rapid development/agile life cycles.” Although HP Fortify has more than 4,000 customers of its application security products (on-premises and cloud-based), we noted that fewer than 500 customers were added in the past year.

■

IBM’s focus on the developer integration leads to exceptional results. The IBM product

offering provides extensive general features on both on-premises and on-demand application security solutions, depending on customer needs. The solution offers limited static analysis features, for data identification, and runtime data tracking. The DAST offering is

well-positioned for web application discovery, Internet-sourced scanning, internal network scanning, and large-scale assessment. IBM has a long lineage in development and has one of the strongest integrations with other product lines and third-party development tools and services. IBM approaches the security market with a developer-centric message and product strategy focus.

■

Veracode unifies the SAST and DAST platform with accurate and timely results. Veracode

offers a unified cloud-based security SAST and DAST platform that includes capabilities for developer workflow integration, central policy management, security analytics and

benchmarking, compliance reporting, and workflow management; differentiators include mobile behavioral analysis and third-party assessments. Veracode has the capability to simultaneously scan thousands of websites through its web perimeter monitoring scanning service while providing high scalability for static binary assessments. With strong results in both SAST and DAST segments, Veracode delivers a high level of accuracy in its technical findings while embracing a customer-centric approach to integration into the greater development workflow

■

WhiteHat Security, although smaller than its competitors, innovates well. Although this

vendor’s dedicated research team is somewhat smaller than those of the other Leaders, the vendor’s overall SAST and DAST capabilities, coupled with its unified SaaS platform, has gained a large customer base ranging from startups to large enterprise companies. WhiteHat Security provides efficient deployment and scalability, frequent scanning updates, and configurable rules and scanning capabilities. While WhiteHat Security offers both SAST and DAST services, its DAST capabilities are more comprehensive and can conduct continuous concurrent scanning of tens of thousands of web applications.

■

Contrast Security is a small vendor making big advancements in application security.

Although this vendor doesn’t neatly fall into either the static or dynamic categories, as outlined in our application security evaluation, Contrast Security provides an up-and-coming offering that makes it a new and innovative contender (since 2012) to the application security space. The Contrast Security solution is instrumented via HTTP requests and responses like a dynamic tool; contains source code and binary code analysis like a static tool; and has an on-server agent like an

(10)

interactive tool. The vendor uses a combination of techniques to provide an effective and detailed analysis for its customers. Contrast Security can autodiscover applications located on a specific server and continually assess the security of these applications in real time. Since the technology is young, there are a few minor configuration options that are lacking; however, the assessment technique is unique enough to warrant interest from forward-thinking enterprise buyers.

■

Quotium innovating in the area of instrumented assessment results in a unique approach.

Another vendor innovating in the space of application security, Quotium has created a solution that doesn’t cleanly fit into the SAST and DAST models of assessment, instead targeting a runtime continuous assessment model by using instrumented analysis. The product’s runtime analysis technology hooks into the application processes and monitors all code execution, while simulating user, and hacker, traffic to the application. Quotium is an ideal product for enterprises looking for simultaneous testing across a multitude of users and servers, with a centralized repository in various test environments (e.g., AWS, Microsoft Azure, and

Rackspace). Customers looking into this solution should note that while Quotium’s Seeker is not a DAST or SAST solution, it does have several unique capabilities that accomplish reasonable dynamic and static results using a unique methodology.

■

Checkmarx delivers SAST directly while offering DAST through partners. Checkmarx’s

solution has strong functional capabilities in deployment, concurrent use, scanning automation, configurable rules and scans, targeted scanning, and multiple user support. General features that the vendor must continue to improve upon include scalability, false positive elimination, and flexible scanning functionality. The Checkmarx offering has strong static analysis features around source code scanning, varied language and framework support, analysis levels, and custom static analysis rules. However, the solution is limited due to an inability to deliver dynamic assessment directly. Instead, Checkmarx looks to partners to deliver the DAST section of its product suite.

strong Performers

■

Beyond Security offers a competitive hybrid option/deployment for its customers.

Customers that require easy deployment across multiple environments should consider looking into Beyond Security as a viable dynamic assessment option. Not only is Beyond Security’s solution available as a self-contained appliance and hosted (cloud) solution, but it also has a hybrid offering (on-site scanner managed by a cloud-based management system) for its customers. Beyond Security’s solution offers competitive dynamic analysis features that support application discovery, Internet-sourced scanning, and internal scanning with either an appliance or VM. However, the solution does not support custom dynamic analysis rules or private data identification. Beyond Security does not offer SAST capabilities in its product suite.

(11)

■

Coverity has a general platform for SAST code analysis but lacks dynamic capabilities.

Coverity’s strength lies in its general features including scanning automation, targeted scanning, multiclass administration, and false positive elimination. Coverity static analysis can analyze byte code for data analysis in Java applications but has little to no functionality in binary scanning and static interactive testing (e.g., code behavior testing and runtime data tracking). Coverity does not support any features to run dynamic analysis.

■

Qualys gets aggressive in product strategy and expands into application security market.

Qualys has traditionally been considered a strong vulnerability assessment vendor, providing continuous security assessment for network-based attacks. Qualys has augmented its technology by moving up the stack into the application assessment realm. The solution offers dynamic analysis features such as application discovery, Internet-sourced scanning, high scalability, and appliance/virtual machine support. Although Qualys’ solution does not support static code analysis, its solution is ideal for customers looking to automate continuous dynamic assessments of target environments. Qualys is gaining in market share and was one of the few vendors to show a significant amount of new enterprise customer growth in the past year. Forrester expects that with new enhancements in its security portfolio, Qualys may become a more direct force in the DAST space in 2015.1

■

Virtual Forge extensively secures SAP-specific content but is limited in other features. Virtual

Forge’s application security solution contains both a cloud-based and on-premises deployment model that has capabilities including concurrent use and configurable rules for source code scanning. Virtual Forge’s solution has limited features available for static analysis testing. The solution analyzes the source code of SAP applications only, limiting the product’s marketability. The offering is unable to fully support static binary scanning, business logic flaws, and interactive testing. Virtual Forge’s product uses open penetration testing frameworks (Metasploit) to do dynamic analysis and scanning, and SAP-specific content is layered on top of these open source tools. Virtual Forge has the most comprehensive product in the space for securing SAP source code; however, you will have to look elsewhere for other languages and features.

Contenders

■

Trend Micro is a new entrant to the market and has some catching up to do. Trend Micro

is a new vendor to the application security market. In Forrester’s evaluation, Trend Micro was one of a few vendors that did not support any SAST capabilities whatsoever, instead focusing only on DAST support from a cloud-only offering. Trend Micro is still developing its product capabilities and strategies and has a robust team with over 1,000 dedicated researchers (including global application security experts) focused on emerging threats and vulnerabilities. The Trend Micro offering is a cloud-based service that can dynamically scale to meet enterprise-level demands but requires some more time in the market before it gains significant traction.

(12)

sUPPLeMenTaL MaTeRIaL online Resource

The online version of Figure 2 is an Excel-based vendor comparison tool that provides detailed product evaluations and customizable rankings.

Data sources Used In This Forrester wave

Forrester used a combination of three data sources to assess the strengths and weaknesses of each solution:

■

Vendor surveys. Forrester surveyed vendors on their capabilities as they relate to the evaluation

criteria. Once we analyzed the completed vendor surveys, we conducted vendor calls where necessary to gather details of vendor qualifications.

■

Product demos. We asked vendors to conduct demonstrations of their product’s functionality.

We used findings from these product demos to validate details of each vendor’s product capabilities.

■

Customer reference calls. To validate product and vendor qualifications, Forrester also

conducted reference calls with three of each vendor’s current customers.

The Forrester wave Methodology

We conduct primary research to develop a list of vendors that meet our criteria to be evaluated in this market. From that initial pool of vendors, we then narrow our final list. We choose these vendors based on: 1) product fit; 2) customer success; and 3) Forrester client demand. We eliminate vendors that have limited customer references and products that don’t fit the scope of our evaluation. After examining past research, user need assessments, and vendor and expert interviews, we develop

the initial evaluation criteria. To evaluate the vendors and their products against our set of criteria, we gather details of product qualifications through a combination of lab evaluations, questionnaires, demos, and/or discussions with client references. We send evaluations to the vendors for their review, and we adjust the evaluations to provide the most accurate view of vendor offerings and strategies. We set default weightings to reflect our analysis of the needs of large user companies — and/or other scenarios as outlined in the Forrester Wave document — and then score the vendors based on a clearly defined scale. These default weightings are intended only as a starting point, and we encourage readers to adapt the weightings to fit their individual needs through the Excel-based tool. The final scores generate the graphical depiction of the market based on current offering, strategy, and market presence. Forrester intends to update vendor evaluations regularly as product

(13)

capabilities and vendor strategies evolve. For more information on the methodology that every Forrester Wave follows, go to http://www.forrester.com/marketing/policies/forrester-wave-methodology.html.

Integrity Policy

All of Forrester’s research, including Waves, is conducted according to our Integrity Policy. For more information, go to http://www.forrester.com/marketing/policies/integrity-policy.html.

enDnoTes

1_{Source: “Qualys Announces Third Quarter 2014 Financial Results,” Qualys press release, November 3,} 2014 (http://www.marketwired.com/press-release/qualys-announces-third-quarter-2014-financial-results-nasdaq-qlys-1963889.htm).

(14)

«

Forrester Focuses On

Security & Risk Professionals

to help your firm capitalize on new business opportunities safely, you must ensure proper governance oversight to manage risk while optimizing security processes and technologies for future flexibility. Forrester’s subject-matter expertise and deep understanding of your role will help you create forward-thinking strategies; weigh opportunity against risk; justify decisions; and optimize your individual, team, and corporate performance.

Sean RhodeS, client persona representing Security & Risk Professionals

informs better decisions, and helps the world’s top companies turn the complexity of change into business advantage. our research-based insight and objective advice enable it professionals to lead more successfully within it and extend their impact beyond the traditional it organization. tailored to your individual role, our resources allow you to focus on important business issues — margin, speed, growth — first, technology second.

foR moRe infoRmation

To find out how Forrester Research can help you be successful every day, please contact the office nearest you, or visit us at www.forrester.com. For a complete list of worldwide locations, visit www.forrester.com/about.

Client SuppoRt

For information on hard-copy or electronic reprints, please contact Client Support at +1 866.367.7378, +1 617.613.5730, or [email protected]. We offer quantity discounts and special pricing for academic and nonprofit institutions.