B UILDING A R OADMAP

Today organizations are performing more tests and more frequently with the hypoth-esis that yesterday’s vulnerabilities were fixed and today there may be another set with which to deal. For companies that practice regular tests, there is an opportunity to collect the information for later analysis.

By performing test after test, security managers gain the necessary information needed to successfully repair holes that represent a threat and establish a baseline for future increased security. In addition, as information is collected over a period of time, trends in the effectiveness to control risk can surface. By investigating the weaknesses and strengths, a well-founded business case for further security invest-ment can be created.

Only a handful of companies have started the practice of managing the data collected from tests for the long-term betterment of the company. By breaking the

previous results into manageable elements, the company’s security officer can iden-tify trends and draw various conclusions on the implemented security controls as opposed to the assumed level of security. This is not always an easy task and one must take into consideration the constant dynamics of security vulnerabilities.

Depicted in Figure 6.1 are the number of vulnerabilities measured without the level of risk identified. Therefore, the figure only represents the effectiveness of the security group to deal with all forms of vulnerabilities.

In this example, a company had tests performed the first week of each month for a year starting in January. The total number of vulnerabilities is the combination of the number of vulnerabilities that were not fixed from the previous test and the new vulnerabilities identified for that testing period.

There are several characteristics worth highlighting. The total number of vul-nerabilities increases initially, declines as the year progresses, and spikes in October.

The spike can be the result of launching a new E-commerce application, Web site, server upgrade, or even something significant such as a merger. Early in the year, the number of vulnerabilities fixed is significantly less than the total number iden-tified. As the year continues, the delta between the two begins to close suggesting the company is getting more efficient at solving problems. Efficiency is typically associated with enacting better processes, such as patch management, integrating tools, or simply adding more resources to perform the work. In the beginning, the inability to fix vulnerabilities quickly resulted in an enormous amount of previously identified holes and then a slight decline as their effectiveness increased over time.

As you can see by the light-grey curve declining over time, this is an average of vulnerabilities that remain from one testing period to the next over the year.

Over time, they reduced the number of total vulnerabilities by increasing their ability to fix them in a meaningful timeframe. Therefore, by the time of a dramatic upward shift in the number of new vulnerabilities late in the year, the company reacted quickly and effectively in short order.

This is representative of a company with very poor security controls early in the year that eventually made the necessary changes to people, processes, and tools to ensure acceptable performance over the long term. In fact, you could assume a new CISO was hired, immediately started having tests performed monthly, and built a team to deal with vulnerabilities, among other security challenges.

Figure 6.1 demonstrates that even the most basic results from tests can be used to support future security-related efforts. Unfortunately, this does not provide enough granular information to address the level of risk for each vulnerability, the overall risk mitigation, or the efficiency of the team to address high-, medium-, or low-rated vulnerabilities. If we recalculate the vulnerabilities by differentiating them by using a weighted value and tracking which vulnerabilities are fixed, we can get more insight as to the activities in addition to the relative state of corporate risk associated with known vulnerabilities.

InFigure 6.2 we introduce the level of severity of each vulnerability and break out which vulnerabilities from each group were repaired or new for the month. With the total number of vulnerabilities, the total fixed, and the total identified from the previous test remaining static, we expose an interesting change in the effectiveness of the security group.

FIGURE 6.1 Determining Effectiveness by Tracking Vulnerabilities and Their Mitigation Road Map

0 2 0 4 0 6 0 8 0 1 0 0 1 2 0

Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New

Jan Feb Mar April May June July Aug Sep Oct Nov Dec

Month

Number of Vulnerabilities

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

Percentage

New Fixed Total

Effective Trend Effective % Previously Identified

FIGURE 6.2 Understanding Overall Security Capability by Adding Risk Measurements to Vulnerabilities Risk Road Map

0 2 0 4 0 6 0 8 0 1 0 0 1 2 0

Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New Total Fixed New

Jan Feb M a r A p r i l May June J u l y Aug Sep Oct Nov Dec Month

Number of Vulnerabilities

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

Percentage

High Medium Low Previously Identifed Effective Trend

Now, we see the number of vulnerabilities fixed were predominantly rated low and medium, with a small percentage of the high-risk vulnerabilities actually being addressed. As each month passed, some new high-risk vulnerabilities were being discovered, essentially digging a “risk-hole” for the security team. By comparing the two efficiency trends we can see the security team is much less effective than first expected. Moreover, because we included the level of risk represented by each vulnerability, the effectiveness trend can be translated to the overall ability to address risk associated with known vulnerabilities.

Detailed in Figure 6.3 are the number of vulnerabilities that are low, medium, and high displayed with the delta between the traditional roadmap and the risk roadmap presented. Towards the bottom of the data are the weighted values associ-ated with the risks. In this example, 15 is assigned to low-rassoci-ated, 30 assigned to medium-rated, and 75 assigned to high-rated vulnerabilities. Of course, any number can be used; however, this scale represents a calculated metric. For example, a medium vulnerability is twice as bad as a low. A high vulnerability is twice as bad as a medium-rated plus a low-rated vulnerability.

Those who have regular tests typically use different providers of the service to ensure the results do not become stale. The byproduct, of course, is that the deliv-erables are different each time, each with a unique format and how information is presented. This adds to the difficulty of normalizing the data to perform a consistent analysis.

Nevertheless, information about the state of security within an organization can be gathered from historical data. For example, if after six penetration tests the number of vulnerabilities associated with Microsoft that has patches is increasing you should revisit your patch management program. If the same vulnerabilities keep appearing over time, you should investigate the existence and use of a standard system con-figuration. Penetration tests are not only an opportunity to test the technical resistance to attack, but can provide insights into the effectiveness of existing management controls. The test also supports and becomes part of the security program to ensure the longevity of security investments and maintain the level of desired security within the organization.

The value of comparing test results can assist with operational demands as well as technical. Companies typically have a secure build, or standard configuration, for systems throughout their network, especially ones exposed to the Internet. Previous test results can be used to further tighten the harness on systems through compre-hensive change management procedures and reinvestigating the standard builds.

Trends in security management can be exposed for good as well as bad practices.

All too often good security practices are implemented and used for a certain period until something comes along to challenge the security program. It can be a new application or service to support a business initiative that was pushed through IT and passed over security. Trends in poor practices surface through identifying similar vulnerability types, such as those relating to applications, protocols, or architecture changes. The evidence can be used to support the argument for more security to not only ensure a secure environment, but to protect future investments.

FIGURE 6.3 Detailed Numbers and Calculations for Risk-Based Roadmap

ROADMAP J a n Feb M a r A p r i l M a y J u n e J u l y A u g S e p

Total # of Vul. 7 3 1 1 0 1 1 3 1 0 6 9 6 8 4 6 4 5 6 4 9

Number Fixed 0 7 2 7 3 2 2 0 1 8 2 8 2 0 3 5

New Vul. 7 3 4 4 3 0 2 5 1 0 6 8 1 2 2 8

Previously Identified 0 6 6 8 3 8 1 8 6 7 8 5 6 4 4 2 1

Effectiveness % 0 . 0 9 . 6 2 4 . 5 2 8 . 3 1 8 . 9 1 8 . 8 3 3 . 3 3 1 . 3 6 2 . 5

Delta % 0 . 0 - 2 . 2 - 4 . 9 - 6 . 8 - 0 . 7 - 4 . 5 - 1 1 . 3 - 1 1 . 8 - 2 5 . 7

RISK ROADMAP J a n Feb M a r A p r i l M a y J u n e J u l y A u g S e p

Total # of Vul. 7 3 1 1 0 1 1 3 1 0 6 9 6 8 4 6 4 5 6 4 9

Low Vul. 3 7 5 4 6 0 5 3 4 3 3 7 2 6 2 1 1 4

Med. Vul. 1 4 2 5 1 7 1 8 2 1 1 5 5 2 5

High Vul. 2 2 3 1 3 6 3 5 3 2 3 2 3 3 3 3 3 0

Total Fixed 0 7 2 7 3 2 2 0 1 8 2 8 2 0 3 5

Low Fixed 0 4 1 2 1 7 1 0 7 1 5 1 0 2 1

Medium Fixed 0 2 1 2 1 1 4 9 1 0 7 7

High Fixed 0 1 3 4 6 2 3 3 7

New Vul. 7 3 4 4 3 0 2 5 1 0 6 8 1 2 2 8

Low New 3 7 2 1 1 8 1 0 0 1 4 5 1 4

Med New 1 4 1 3 4 1 2 7 3 0 4 1 0

High New 2 2 1 0 8 3 3 2 4 3 4

Previously Identified 0 6 6 8 3 8 1 8 6 7 8 5 6 4 4 2 1

Effectiveness % 0 . 0 7 . 4 1 9 . 7 2 1 . 5 1 8 . 2 1 4 . 3 2 2 . 0 1 9 . 4 3 6 . 8

Wt. VALUE J a n Feb M a r A p r i l M a y J u n e J u l y A u g S e p

W e i g h t Total Wt. Total Wt. Total Wt. Total Wt. Total Wt. Total Wt. Total Wt. Total Wt. Total Wt.

1 5 5 5 5 8 1 0 9 0 0 7 9 5 6 4 5 5 5 5 3 9 0 3 1 5 2 1 0

3 0 4 2 0 7 5 0 5 1 0 5 4 0 6 3 0 4 5 0 1 5 0 6 0 1 5 0

7 5 1 6 5 0 2 3 2 5 2 7 0 0 2 6 2 5 2 4 0 0 2 4 0 0 2 4 7 5 2 4 7 5 2 2 5 0

Total 2 6 2 5 3 8 8 5 4 1 1 0 3 9 6 0 3 6 7 5 3 4 0 5 3 0 1 5 2 8 5 0 2 6 1 0

W e i g h tFixed Wt. Fixed Wt. Fixed Wt. Fixed Wt. Fixed Wt. Fixed Wt. Fixed Wt. Fixed Wt. Fixed Wt.

1 5 0 6 0 1 8 0 2 5 5 1 5 0 1 0 5 2 2 5 1 5 0 3 1 5

3 0 0 6 0 3 6 0 3 3 0 1 2 0 2 7 0 3 0 0 2 1 0 2 1 0

7 5 0 7 5 2 2 5 3 0 0 4 5 0 1 5 0 2 2 5 2 2 5 5 2 5

Total 0 1 9 5 7 6 5 8 8 5 7 2 0 5 2 5 7 5 0 5 8 5 1 0 5 0

Eff % 0 . 0 0 7 . 4 3 1 9 . 6 9 2 1 . 5 3 1 8 . 1 8 1 4 . 2 9 2 2 . 0 3 1 9 . 4 0 3 6 . 8 4