WHAT CONSTITUTES A SUCCESS?

Given that this book is focused on the value of a test, the definition of a successful attack is not only a constant theme throughout the material, but, as we show, it can be much more than simply the systems that were hacked. This is an opportunity to introduce the primary characteristics of a test that can be used to evaluate the overall success of an engagement.

The definition of a successful test can be elusive. Much of a test’s success or failure is founded on the goals and objectives stated at the onset of the test. To state the obvious, without planning and some form of goal, there is little chance of determining what was actually accomplished.

There are many metrics that can be employed to rate the success of a test, but the most predominant one is technical exploitation. Having a tester penetrate an online application and gain access to a database of credit card numbers has significant tangible characteristics, which are therefore easy to measure.

Another aspect of a success can be the management of the test. For example, how well was the test conducted? Many organizations establish operating parameters to protect systems, employees, and customers from any potential threat that may come from hacking systems. The most obvious is downtime. Bringing a business- critical system down in the middle of the business day can be a costly mistake. How the information collected about the target handled (e.g., protected) during the test will certainly be scrutinized. If the list of vulnerabilities and how they were exploited were to become public, the test would move quickly from success to damage control. Some organizations base the success of the test on the deliverable. The quality of the deliverable is paramount to many, understandably so, and even in cases of total technical failure, the deliverable can substantiate a success.

The interchange of value and success will occur in every test. Typically, the definition of success will be associated with meeting a set of specific goals. More often than not, these goals are those vulnerabilities that are identified and successfully exploited. This should come as no surprise because the foundation of the test is typically to hack a target! However, even the exploitation of a vulnerability does not constitute a success. In fact, in some cases, exploiting a hole is exactly what the target does NOT want and success is founded on what can be identified—not broken. On the other hand, there are companies that insist on evaluating the exposure to attack and are only satisfied if the vulnerability is exploited. Typically, this demand is associated with a specific target, such as a new application, change in the infra- structure, or the addition of new untested technology. Nevertheless, there are many situations where the goal is simple—gain access—and not to accommodate the demand is grounds for failure no matter how well the test was managed, the deliv- erable quality, or the execution.

NOTE1: DIGGING FOR THE HOLE

In a meeting with a long-term customer that has monthly tests against their Internet-facing infrastructure, a concern for the potential for someone to hack into their remote access solution was questioned. Up until this point, the success of the test was heavily placed on the deliverable and the identification of vulnerabilities—not exploiting any holes. They preferred to know what the problems were and have us recommend fixes as opposed to potentially causing harm.

In contrast, the next test was to exploit any vulnerabilities in the remote access solution and gain as much information and access as possible. An aggres- sive test was planned and performed shortly thereafter. The tester gained access to the terminal server (Citrix) by circumventing the poor integration of the Web application, but could not exploit any opportunities to gain access to back-end applications published by the Citrix system.

The result was considered a failure, which was interesting given that all previous tests were based on validation and identification of problems and the quality of the deliverable. Nevertheless, one has to agree with the conclusion. The goal was set, objectives defined, and scope determined, and the target was not met.

Later it was confided by the client that success was expected based on our tester’s familiarity with the environment and the remote access solution, which had been in place for over a year. Although knowing a target does not imply success, the point was valid.

Technical attributes of the test are commonly used as the measuring stick for success. As mentioned above, when someone exploits a vulnerability and obtains

valued data the vulnerability is defined as well as what was performed to gain access. Both of these elements go a long way in fixing the problem. Therefore, the test’s results can be employed and acted upon to reduce future potential harm.

The value of the test is more convoluted, open to more interpretation, and can exist even in the light of a defined failure. If a company seeks to have a new custom application tested and exploited to evaluate the security features of the code, the test may not be considered successful if nothing is exploited. However, the value to the organization may still exist. The value can be as simple as knowing the application was tested and now the company can feel confident in deploying or moving to the next phase of development. Or, the value can be the raw data that was collected by the tester and the tools used to gain more insight into how the application responds to different tactics.

To add to the malaise, the reality is that usually, somewhere in the process, someone is not going to be happy with the test and, depending on who that person or group is, can sway the interpretation of success and most certainly value. The internal politics of an organization can be very convoluted and when a third party is brought in to perform a test it can be the seed of future contention. The admin- istrator of a server that was compromised may argue the test’s validity because he is now in the spotlight. It is not uncommon to have entire departments lash out at the test’s results because someone else initiated the test and the results were not favorable for them.

Finally, there is the consultant’s perspective. If the tester does not exploit any vulnerabilities as demanded by the customer, but the client feels the test was a success, that does not mean the consultant feels the same way. In fact, I know of no tester who wouldn’t feel disheartened in some way and begin to question her tactics. It is almost commonplace to talk to disappointed consultants even after a successful test; it is part of a tester’s mentality to overachieve and push the limits of the target as well as herself. It is important to consider the consultants’ perspectives of success and ensure there is the foundation for future success by their definition. This can be accomplished by training, shadowing on other engagements, or allowing them to focus on tests that require their core skills. From a service provider’s point of view, it is important to consider both the client’s as well as the tester’s feeling of success because both will affect the future of the business.