Part 3: Performance Testing

Part 3: Performance Testing

Author – Bob Walder

Overview

Security vendors’ marketing claims are often exaggerated and frequently do not reflect real-world or enterprise conditions. The performance of security devices is difficult to determine accurately, and yet failure to do so can result in significant negative impact on the network should the wrong device be selected, or should a device be configured incorrectly.

Security professionals must be prepared to test security products to confirm their security effectiveness and performance capabilities under real-world conditions. Testing can expose performance-related problems caused by inappropriate security products, including high latency and frequent “fail closed” events. This in turn may result in active devices being deployed passively or in blocking being disabled, making the devices significantly less effective

Testing extremes of performance in a live network is unwise, and thus an isolated test bed is recommended. High-performance test tools can be purchased or rented to enable organizations to perform the full range of

performance tests with realistic traffic and user loads, and a fully loaded test environment can be contained within a single equipment rack.

Testing is not necessarily about proving that the most capable, most expensive model is the best choice. A well-designed testing plan may show that a lower level of performance is acceptable at certain points on the network (in a branch office deployment, for example), and this can reduce your purchase and deployment costs. IT organizations that do not perform relevant tests in-house may overspend significantly on performance and coverage that their companies do not need.

This Best Practices document is part of a series on testing security products, and in particular should be read in conjunction with its companion, Enterprise Security Product Testing for the Enterprise: Best Practices, Part 2: Security Effectiveness Testing.

NSS Labs Findings

 The device’s location within the network will affect its performance; therefore, it is essential to determine the correct use case for the device under test (DUT) and create an appropriate methodology.

 The aim of testing is to determine how a device performs in your specific environment, rather than to validate the vendor’s own performance figures.

 Testing with the security policy appropriate to your intended use case implies that all appropriate security modules and features are enabled to support this, which will usually result in a maximum throughput far lower than that claimed by the device manufacturer.

 When an inline device placed in a live network performs poorly, this can have serious consequences as latency increases to unacceptable levels. High latency or frequent “fail closed” events can result in active devices being redeployed in a passive state, or in blocking being disabled, which significantly reduces their effectiveness

 Without performing relevant tests in-house, organizations could be persuaded to overspend significantly, purchasing devices with performance and coverage levels that are not required

 A complete test lab can be created in a single rack of equipment, making it possible for almost any organization to perform in-house testing.

NSS Labs Recommendations

 NSS Labs subscribers should make extensive use of available NSS resources (Test Reports, Product Selection Toolkits, Market Analyses, and analyst inquiry) to evaluate overall performance under extreme conditions and to provide a starting point for their own tests.

 Determine the required use case, and define testing criteria based on that use case.

 Ensure that sufficient headroom is built into the performance capabilities to enable the device to handle any significant changes that may occur over its projected life.

 Take advice from NSS where in-house skill sets may be lacking in key areas such as methodology creation, test configuration and execution, and results interpretation.

 The use of high-performance, dedicated hardware-based test tools is recommended for the best results. Where outright purchase is infeasible, investigate leasing or short-term rental options.

 Implement continuous testing initiatives to ensure performance remains consistent across updates of firmware and signature/software packs.

 Evaluate prepackaged test packs from test tool vendors such as Ixia, which have been designed to create a replica of the performance tests run in-house by NSS engineers.

Table of Contents

Overview ... 1

NSS Labs Findings ... 2

NSS Labs Recommendations ... 2

Analysis ... 4

Create a Test Plan ... 4

Seek Out Sources of High-Quality Independent Product Testing and Advice ... 4

Determine the Use Case ... 5

Define Evaluation Criteria... 5

Select Appropriate Test Tools ... 7

Configure the Test Tools ... 8

Average Packet Size ... 9

Typical Protocol Mix ... 9

Configure the Device Under Test ... 9

Run the Test ... 10

Evaluate the Results ... 11

Beware of Unethical Practices ... 12

Calculate Total Cost of Ownership ... 12

Scrutinize Vendor Track Record ... 13

Updates ... 13

Accuracy ... 13

Carefully Review “Awards” and “Validation Reports”... 13

Implement Continuous Testing Initiatives ... 13

Reading List ... 14

Analysis

Performance testing is a critical component of any in-house competitive evaluation. An inline security device is expected to be reliable (not crash), to never block legitimate traffic, and to not unduly affect network

performance. Testing must be performed both under normal traffic loads and under more extreme loads to ensure performance does not degrade.

Independent third-party performance tests, such as those performed by NSS engineers, push devices to the extreme in order to find all of the “corner cases” and to ensure that they can hold up in the widest possible choice of deployments. A typical corporate network, however, will consist of a different traffic mix, different average packet sizes, and different average load levels. It would be rare to find a network that pushed devices as hard as NSS does in its testing, and you may well find that your own environment is considerably less demanding. This could allow you to save money by purchasing a lower specification device than you originally thought necessary, especially in less-critical areas such as branch office deployments.

Do, however, pay particular attention to the maximum performance and latency of a given device. Absolute performance limitations and latency are difficult to ascertain in a live network, and vendors may use this uncertainty to pitch lower-cost devices, which may cause problems in the long term.

Accurate performance testing of network security devices is complicated, and those considering purchase of these devices would be advised to seek expert help. Once again, NSS is well placed to offer advice and assistance to clients, as are the vendors of the high-performance hardware-based load generation tools used in this type of testing.

There are a number of key steps that should be followed to ensure success when evaluating the performance of security devices:

Create a Test Plan

Creation of the test plan is the most crucial step in selecting a network security product. An effective test plan enables the enterprise to select cost-effective security solutions that align with internal requirements for performance and system integration.

IT organizations that do not perform relevant tests in-house may overspend significantly on performance and coverage that their companies do not need. Creation of a test plan should therefore be the first step in any product evaluation and should be closely correlated to the business and functional requirements of the enterprise. For more on creating a test plan, see Part 1 of this Best Practices series on Enterprise Security Product Testing. Seek Out Sources of High-Quality Independent Product Testing and Advice

NSS subscribers should make extensive use of the numerous NSS resources (Test Reports, Product Selection Toolkits< Market Analyses, and analyst inquiry) available to them as part of their subscription services. Vendors that have never submitted their products for such tests or that rely purely on magazine reviews to market their wares should be treated with caution. It is possible that they have concerns about the performance of their own products, or that they simply do not understand the value of testing in their own QA processes.

Independent group test reports can be used by the security professional to create a shortlist of products to be tested. This can reduce the time and cost associated with testing by eliminating unsuitable devices prior to any

tests being run. These reports can also be used to identify devices that demonstrate higher levels of performance than those claimed by the vendor. This can often happen with devices at the lower end of the market, which share common hardware and code base with their enterprise-ready cousins. When looking for a device with a minimum throughput of 500 Mbps, for example, it would be easy to dismiss devices rated at less than this by the vendor. However, in one of NSS’ early intrusion prevention system (IPS) group tests, one product rated at 350 Mbps by the vendor actually performed at almost 800 Mbps with a tuned policy. Given that it was priced as a 350 Mbps device, the potential cost savings in a large multisensor deployment would have been significant.

IT organizations can also save significant time and money by starting with and then modifying one or more of NSS’ extensive test methodologies, which are free and available to everyone. While these methodologies will rarely be suitable immediately for your own environment (test labs have to test extremes and “corner cases”), the basic types of test described within can be adapted to your own needs.

Determine the Use Case

Determine exactly how you want to test the devices on your shortlist. Do not underestimate the importance of determining accurately the intended location and traffic profile of the device to be tested (see Part 1 in this series of Best Practices documents on Enterprise Security Product Testing).

For example, an NSS group test may have commented adversely on the ability of a device to support high levels of TCP connections per second. However, if you are not intending to protect a large server farm, this may be of little concern. Performance in some areas can be sacrificed for more important features, including higher security effectiveness or lower price point.

Define Evaluation Criteria

It is often difficult for a security administrator to be able to characterize the traffic on his network with a high degree of accuracy, but this is a necessary precursor to running a successful test. By answering a few critical questions, you can acquire valuable insight into the required evaluation criteria:

 What is the average bandwidth?  What is the peak bandwidth?

 Is the traffic to be inspected primarily composed of one protocol, or is there a mix?

 What is the average packet size and level of new connections established every second? Both of these are critical parameters with regard to the performance of inline deep inspection engines, for example. If your hardware is operating at its design limits, all of these questions need to be answered as accurately as possible in order to prevent performance degradation once the device is deployed. The alternative is to play it safe and select a product that is clearly overengineered for your particular environment. This, however, will have an impact on the cost. Security administrators must also anticipate how the above requirements could change over the lifetime of the device once it is deployed, and they must ensure that the device has the headroom to handle future capacity needs.

A security device is expected to be reliable (not crash), to never block legitimate traffic, and to not unduly affect network or host system performance. Testing must be performed under normal traffic loads, and under more extreme loads to ensure performance does not degrade.

Packet processing and TCP connection rates must be measured at the rated speed of the device under real-life traffic conditions, and the device must meet the stated performance with the appropriate security policy applied.

Testing with the security policy appropriate to your intended use case implies that all appropriate security modules and features are enabled to support this. This will usually result in a maximum throughput far lower than that claimed by the device manufacturer, which will often test with an optimum traffic profile and security effectively disabled. For example, an NGFW vendor will frequently quote firewall-only statistics in marketing materials, and often with large packet sizes. The results will therefore be largely irrelevant to an environment with “average” packet size that requires firewall, IPS, and web content filtering. The aim of in-house testing is to determine how the device performs in your specific environment, not to validate the vendor’s own performance figures.

Headroom should be built into the performance capabilities to enable the device to address any changes in policy (which might require processing additional signatures) and any increases in the size of signature packs that may occur over the projected life of the device. Ideally, the detection engine should be designed in such a way that the number of signatures or rules loaded has no adverse effect on the performance of the device, and your tests should be designed to demonstrate this.

Stability and reliability are also key. Some devices may perform well under high levels of attack traffic for a short period of time but then begin to degrade or even fail altogether. Stateful devices must prove that they can maintain state even as the number of connections approaches (or exceeds) the maximum specified by the vendor. Security products should be evaluated against enterprise-specific criteria, which are then used to determine test tool configurations, as well as provide benchmarks against which the test results can be evaluated. Typical examples include:

Acceptable Packet Loss

While the idea of packet loss is unpleasant, the reality is that most network protocols are capable of recovering from occasional losses. Instead of demanding perfection (which is unrealistic), you may choose to accept limited packet loss when sending multi-Gigabit traffic composed purely of 64-byte packets, while imposing a zero-packet-loss criterion for your average packet size.

Acceptable Latency

Latency (the time it takes for a packet to pass through an inline device) is important in any network environment, and it is the single metric that is most likely to bring the security team into conflict with the network team. Latency may impose an upper bound on throughput, and it also has an impact on interactive applications, thus affecting user response time. The latency and throughput of an inline device must be on par with other equipment in the network in which it is deployed. An inline security product, for example, must strive to perform much more like a switch than a typical passive security device, especially when it is necessary to install more than one appliance in the same data path.

The latency of the DUT must be evaluated in the context of the network in which it is deployed. This can range from one to two milliseconds for Internet-connected perimeter solutions through one millisecond or less for MAN/WAN links, and just a few hundred microseconds for core network deployments.

Latency is often subjective and can be almost irrelevant if you are talking about a single device to be installed at the network perimeter. However, once you require two or three devices to be installed serially in the network core, that hundreds-of-microseconds latency that the vendor assures you is “perfectly OK” suddenly becomes an unacceptable delay.

Acceptable Connection Rate

Every time a user connects to a web server, the inline security device must create a connection of its own to monitor the session for malicious traffic. These connections must be managed effectively and torn down once the session has ended— and connection management is the most resource intensive process an inline security device has to perform. If your web server farm is capable of supporting 200,000 new TCP connections per second (CPS), then the inline security device protecting it must be able to match, and preferably exceed, it.

Naturally, the size of the HTTP response has a bearing on the number of connections that can be made each second—the smaller the response size, the higher the connection rate that must be supported. This is determined by the web application and should be modeled carefully in advance of testing to obtain the most accurate results.

Acceptable User Response Time

User response time is a function of latency, connection rate, and the back-end processing required to support the transaction being performed.

Although user response time is often somewhat subjective, you should have already determined maximum acceptable response times as part of your application development life cycle. These should be part of the test criteria.

Acceptable Behavior When Device Resources Are Exhausted

It is important to determine (either through testing or by questioning the vendor) whether a security device will fail closed (block all traffic) or fail open (allow all traffic without inspection) as resources are exhausted (for example, as the device runs out of memory to track new sessions).

This will either result in new connections being refused (causing denial of service) or in the oldest connections being discarded (providing potential evasion opportunities). There is no right or wrong approach, but what is important is that the device’s behavior can be configured to either option—depending on the requirements of your environment. The inability to configure this behavior may eliminate certain devices from the selection process.

Select Appropriate Test Tools

It is not a straightforward process to accurately test the performance of security devices, and those considering their purchase would be well advised to seek expert help. Fortunately, the physical testing requirements are less onerous today with the introduction of high-performance, hardware-based load generation tools from vendors such as Ixia and Spirent Communications. However, simply plugging these devices into your network and pressing “Go” is not an acceptable approach.

In the absence of in-house expertise on these tools, vendors will usually offer on-site support and consulting. NSS subscribers should engage with NSS analysts directly in order to make use of their expertise. NSS analysts can advise not only on configuring and operating the test tools in your network, but also on creating a solid test methodology and—most importantly—on correctly interpreting the final results.

Hardware-based test tools are typically used to simulate hundreds of gigabits per second of real-world network traffic from millions of users in order to test performance through the DUT. Some, however, will also provide the ability to launch malicious traffic, either alone or as part of a typical network traffic mix. The ability to do this at wire speed makes these tools useful for DDoS testing in addition to basic security effectiveness validation. If you

are fortunate enough to have these devices at your disposal for performance testing, do not ignore their potential usefulness as security effectiveness testing devices, too.

These tools can be hired if purchase is out of the question, and when budgeting, it is important to recognize that they can be used for more than security performance testing alone. They are equally adept at stress testing network equipment such as switches, routers load balancers, bandwidth management devices, and even servers— in fact, that was their original function.

One low-cost alternative to dedicated hardware-based test tools would be to use open-source replay tools such as

tcpreplay with large packet capture files from your own network. However, this is an imperfect solution that

cannot provide fine-grained control over the traffic generated, nor can it reach the levels of performance provided by dedicated hardware-based test tools. This option should be selected as a last resort only for ad hoc and low-performance testing projects—be wary of test results presented by vendors and test labs that have used this method.

Figure 1 – Basic Test Environment

Where cost restrictions make it impractical to create an in-house test environment, it is still important to evaluate potential purchases by using external testing resources (NSS subscribers can discuss the possibility of using the NSS test lab to perform competitive tests) or by installing test products in a live network. In the latter case, however, it should be recognized that the type of testing available would be more restricted in order to ensure there is no adverse impact on business processes.

Configure the Test Tools

In any test, the most accurate results will always be returned if the traffic in the test environment mirrors as closely as possible the traffic in the live network at the location in which the device will eventually be installed.

When using hardware-based performance test tools it is tempting to employ some of the canned configuration scripts included. However, this approach is ill-advised because you will not know exactly what the test tool is doing, and the results can often be misleading.

Use your use case and test criteria to determine what type of traffic should be generated to replicate the deployment conditions as closely as possible, and then create testing configurations to reflect that.

Where possible, you should install network monitoring equipment in the network at the same location in which the DUT will eventually be deployed, and you should measure your network traffic over an extended period of time to determine critical performance parameters such as:

Average Packet Size

The average packet size on a network or in a test environment has a dramatic effect on the performance of an inline security device—small packets (64 bytes) are extremely difficult to process at wire speeds on multi-Gigabit networks, whereas large packets (1500 bytes) are much easier to handle.

NSS engineers will normally use in the region of 440-550 bytes as a typical “average” packet size in the key stress tests, though this will vary considerably (up and down) in the various real-world tests depending on the target environment being simulated. At the other extreme, some vendors will use 1,500-byte packets in their tests to demonstrate efficient packet processing at high speed. Your own network may offer up something completely different, and the average packet size typically will change in different parts of the network—the network core will have a completely different traffic profile to a server farm in the DMZ, for example.

Use the network monitoring statistics to determine the typical mix of packet sizes and the overall average on your own network, and then endeavor to replicate this mix in your test configurations.

Typical Protocol Mix

As with the packet size, the mix of protocols and content on the network at the point of deployment will also have a significant impact on the performance of the DUT. For example, if the majority of the traffic is voice, an inline inspection engine will have far less work to perform than if it were mainly HTTP.

The content is also important. As another example, the HTTP response size will affect the connections per second and transactions per second results, as will the data returned in the HTTP response.

Use the network monitoring statistics to determine the typical protocol mix in the traffic at the point of deployment and then endeavor to generate realistic content (actual web pages and DNS responses) when replicating this mix in your test configurations.

Configure the Device Under Test

The first pass of testing should always be performed with the vendor’s default or “recommended” policy applied to determine its efficacy in your environment. Does the policy include coverage for all of the traffic types you see on your network? The chances are that it will not, and that tuning will be required that will affect the overall performance of the DUT.

After running initial performance tests or deploying the device on your live network, you may find that further tuning will be required in order to eliminate false positive alerts (signatures that are triggering incorrectly based on legitimate traffic in your network.) Once that tuning has been completed, it will be necessary to run additional

security effectiveness and performance tests to make sure you did not break anything in the security policy. This cycle should be repeated until you achieve the optimal combination of maximum performance with maximum security effectiveness and minimum false positives.

Configuration of the DUT should always be performed in the first instance with the aid of the vendor’s engineers. Make them aware of your use case and test criteria and ensure that their configuration precisely reflects your needs. Some devices will be shipped with critical security features disabled for performance reasons—anti-evasion capabilities are an obvious example; server-to-client signatures are another.

Be sure to verify that no such features are disabled prior to running your performance tests. This can be most easily achieved by running security effectiveness tests first. (See Part 2 of this Best Practices series on enterprise security product testing, which covers security effectiveness testing.) If key coverage tests fail, you have evidence that can be used to challenge the vendor to ensure that the device is configured as required by your test plan.

Run the Test

Run multiple performance tests with no policy (or with an empty policy) applied as a baseline, then with the vendor’s default or recommended policy, and finally with your own security policy as determined by your use case and selection criteria.

The differences between default and tuned security policies are often significant, which is why it is so important to perform your own testing in-house. Vendor tests will usually favor their “best case” scenarios and will often be run with background traffic that is not representative of a real-world network—often with no security policy applied. Performance can increase as well as decrease, however, depending on the type of signatures required in your own policy.

During the test, look for devices where the difference in performance between default and tuned policies is minimal, since this demonstrates an efficient inspection engine design that will not be unduly affected by the number of signatures applied. Also, look closely at the differences between the vendor’s claimed performance and actual performance as measured by you.

The best results will always be provided by a test that is run using real traffic on your live network. However, it is rarely possible to vary the amount of traffic in a controlled manner on a live network, and testing a device to destruction means crippling your network at the same time. It is for this reason that you should invest in a dedicated test network, or in a test environment such as that depicted in Figure 1.

You can use NSS group test reports to discover all the corner cases that extreme performance testing can highlight, and it is not necessary to replicate those tests. However, it is still important that your own tests reflect both normal network traffic and extreme conditions that may be experienced at times of peak load. For example, if you are protecting an e-commerce site for a store that has a low price promotion one day per month, your security devices must be able to handle the peak loads of that day without causing an adverse user experience. Run tests that are designed to exercise all aspects of the security device according to the evaluation criteria defined earlier. If your criteria state that 200,000 connections per second is the minimum requirement with a 20-Kbyte HTTP response in order to support your e-commerce server farm, then test beyond this threshold to determine the point at which the device begins to fail. Repeat this with each of the key evaluation criteria you have defined.

Stability and reliability are also critical. Having obtained the key performance figures, any devices which remain on your shortlist for purchase should be subjected to lengthy (weeks, if possible) deployment under severe test conditions including normal, extreme, and malicious traffic profiles. Some devices may perform well under high levels of attack traffic for a short period of time, but then begin to degrade after a while, or even fail altogether. Following this testing, there should be a period of deployment in your live network to ensure that your live traffic does not contain anything that may cause unexpected failure or serious false positive alerts. At least part of the evaluation for any inline security product should be performed in full blocking mode. Do not make a major purchasing decision based on an evaluation where the device has been in detect-only mode throughout the test, even if it was operating inline. It is not unheard of for devices that work perfectly well in detect-only mode to fail once blocking is enabled. This can be disruptive to your network if it occurs when the device is deployed in a live environment rather than an isolated test environment, but it is preferable to discover this “feature” before spending hundreds of thousands of dollars on a multisensor deployment, only to discover that you have purchased several very expensive passive IDS sensors that cannot be used inline.

Resist the urge to shorten the test cycle by installing multiple devices inline simultaneously, since one device can affect the performance and effectiveness of the others. Always try to deploy DUTs individually at the same point in the network. If it is necessary to simultaneously deploy all devices inline for the extended reliability testing, ensure their positions are rotated during the test. Some devices will perform “protocol scrubbing” or other traffic

normalization, which will leave the devices behind them free of malformed traffic that might otherwise have caused them problems when placed first in line.

Evaluate the Results

Although many of the performance testing tools give the impression of having “plug and play” capabilities, simply plugging in and pressing the “Go” button rarely produces useful results. The output invariably will be an impressive looking graph, but interpreting that is quite an art, which is made all the more difficult if no information is provided on what the test was actually designed to accomplish.

Part of the process of defining the test plan was to describe the type of traffic mix (for example, protocols, and average packet sizes) required to stress the DUT in the most appropriate way. Part of the skill in interpreting the results is to determine at which point the DUT begins to fail gracefully, and at which point it fails completely. At which point do you start to notice excessive delays, dropped connections, and a generally unacceptable user experience?

Set “breaking points” for each of the tests in your testing plan and stick to them. For example:

 What level of latency are you prepared to accept? 300 microseconds? Sub millisecond? Greater than one millisecond?

 What is the minimum packet size where you require 100 percent detection rates? Is 64-byte packet

performance important to you? How about 128 bytes, or 256 bytes? Be realistic; insisting on 100 percent line-rate detection speeds with 64-byte packets will demonstline-rate engineering excellence, but it will come at a significant price, and it is probably irrelevant in most enterprise environments (service providers may have a different perspective).

 What level of TCP connections per second (CPS) do you require to support normal operation of the critical assets you are protecting? HTTP CPS? HTTP transactions per second (TPS)? SMTP TPS?

 What level of outstanding TCP connections are you prepared to accept before determining that a test has failed?

 What level of user response time are you prepared to accept before determining that a test has failed?  Is there any level of traffic where you will accept a fail-closed or fail-open result?

 How many stateful connections do you require the device to support before it rejects new connections or ages out old ones?

It is also important to recognize when your test is not stressful enough—does the device have more than enough headroom or did you not generate enough load? Ensure that headroom is sufficient to accommodate changes in network traffic that are likely to occur throughout the life span of the product.

After several successful test runs, it soon becomes second nature to extract the critical breaking point from a mass of Microsoft Excel spreadsheets and line graphs, but initially this can be a challenge. This is another area where you can make use of external resources, and where NSS subscribers should engage directly with our analysts, to help with your initial analyses.

Although you may be under time constraints, do not resist deviating from the test plan if you identify serious issues. Working with a vendor to identify problems with its product may help improve the product and ensure it is suitable for your environment.

Beware of Unethical Practices

Be on your guard against unethical practices from vendors when testing their products against those from competitors. Almost every security device has a weakness of some kind that will never normally manifest itself on a normal network. There is always a trade-off between performance and security, and the real difference between many of the products on the market is where that trade-off has been made in the architecture of the device. However, some vendors have been known to craft “special” packet captures or custom utilities to directly exploit the specific architectural shortcoming of a competitor’s device, and show it at its worst. Ignore such

demonstrations, and consider whether a vendor with such serious ethical issues is one with whom you wish to work. The real test, as has already been stated, is how the device performs in your network with your traffic.

Calculate Total Cost of Ownership

The final stage in evaluating the results should be to calculate total cost of ownership (TCO) over the expected life of the product.

Update services can be costly, and TCO calculations should include the cost of such services both in the first year (when the cost is often rolled into the initial purchase price) and in subsequent years. Where multifunction devices are deployed, it is important to factor in update services for each security module (for example, IPS, content filtering, and antivirus).

The cost of day-to-day management and regular tuning should also be factored in. Where devices are operating at their design limits, and where administrators are forced to aggressively tune devices to achieve the required performance, the ongoing management costs will be higher than for devices with sufficient headroom built in. The same is true of devices that are prone to high levels of false positive alerts. Such aggressive tuning is also error prone, making it more likely that underspecified devices will leave critical network assets unprotected, risking compromise and resulting in loss of customer confidence and possible loss of compliance status.

Scrutinize Vendor Track Record

No matter how promising a device appears during testing, it should be recognized that vendors are never so responsive as they are during the sales cycle. Prospective customers should read NSS research and speak to NSS analysts and vendor customers (current and previous, if possible) to determine key points regarding a vendor’s track record:

Updates

Ensure that the vendors you have shortlisted have adequate resources committed to addressing serious performance and stability issues with their products.

The security vendor must be able to respond quickly to offer a rapid work-around, followed by a targeted update/patch.

Accuracy

Perform research to determine a vendor's track record with false positives. How often do signatures have to be revised after initial release?

Carefully Review “Awards” and “Validation Reports”

Study the claims of independent validation reports and awards carefully. If all a vendor can show you is a Microsoft PowerPoint slide full of magazine “awards,” this indicates a lack of understanding of the need for good testing, or a lack of confidence in the product. Make sure the methodologies used in independent reports are sound, and make sure that the testing house has a pedigree in security testing specifically.

Implement Continuous Testing Initiatives

Do not restrict testing to the purchasing cycle alone—it should be an integral part of your ongoing security maintenance regime.

NSS has observed instances where a single poorly-written signature has crippled the performance of an inline protection device. Firmware updates can also break previously solid inspection processes—anti-evasion techniques appear to be particularly prone to disruption between firmware updates.

Following initial deployment, perform a full benchmark test to acquire a baseline. Then, each time you apply a new firmware update, signature pack update, or change in security policy (however minor), the device should be retested and the results compared against the baseline. This way, it is possible to monitor and correct detrimental changes in performance or security effectiveness.

It is always preferable to retest devices in identical conditions to the prepurchase and baseline tests—usually in an isolated test environment. This, however, would require a duplicate device or the ability to swap out high

availability (HA) components temporarily. Where this is not feasible, hardware-based test tools can be used to generate fixed loads of low-level background traffic on a live network to provide an indication of latency and user response times.

Reading List

Security Product Testing for the Enterprise, Part 1: Fail to Plan, Plan to Fail. NSS Labs

Enterprise Security Product Testing: Best Practices, Part 2: Security Effectiveness Testing. NSS Labs Enterprise Security Product Testing: Best Practices, Part 4: Management. NSS Labs

NSS Labs Product Selection Guides NSS Labs Market Analyses

© 2016 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, e-mailed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”). Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these conditions, you should not read the rest of this report but should instead return the report immediately to us. “You” or “your” means the person who accesses this report and any entity on whose behalf he/she has obtained this report.

1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it. 2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever arising from any error or omission in this report.

3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption.

5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.

6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners.

Contact Information

This report was produced as part of NSS Labs’ independent testing information services. Leading products were tested at no cost to the vendor, and NSS Labs received no vendor funding to produce this report.