Evolutions in Browser Security
TRENDS IN BROWSER SECURITY PERFORMANCE
Author – Randy Abrams
Overview
This analyst brief aggregates results from NSS Labs tests conducted between 2009 and 2013 in a comparison of phishing and socially engineered malware (SEM) protection by the leading browsers. Figure 1 reveals trends in protection levels of the four leading browsers, comparing combined test results from 2009 to the recent 2012 and 2013 scores.
Figure 1 – Leading Browser Malware and Phishing Block Rates (2009, 2012, 2013)
2009
2012
2013
2009
2013 2009
2012 2013
2009
2012
Av er ag e Ph is h
Average Malware
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 20% 40% 60% 80% 100%
Ma lw ar e
Phishing
Chrome Firefox IE Safari
Average Phish
Average Malware
In the NSS tests, the browsers are rated on performance in four categories:
• Average phishing block rate
• Zero-‐hour phishing block rate
• Average SEM block rate
• Zero-‐hour SEM block rate
Internet Explorer (IE) shows a consistently superior ability to block SEM, while providing competitive phishing protection; it leads the tested browsers in combined protections for these categories. Google’s Download Protection technology has improved significantly over time, placing it behind IE but well ahead of Firefox and Safari. Both Firefox and Safari lead the other browsers in phishing protection but provide negligible protection against SEM attacks.
This analyst brief includes data from previously published NSS phishing tests conducted in 2009, 2012, and 2013, as well as SEM tests published every year from 2009 to 2013. In 2010, two SEM tests were published.
Figure 2 shows the overall performance of the browsers for the 2013 browser phishing and SEM tests.
Figure 2 – 2013 Combined Phishing And Malware Block Rates
Figure 2 treats all protection metrics in 2013 equally. Later in this analyst brief, Figure 15 will add weighting based on the relative protection importance of the tested parameters in order to provide a more realistic ranking of the browsers in 2013. Figure 14 provides an aggregation of all of the tests from 2009 to 2013, with emphasis placed on freshness and relative importance of the test metrics.
52%
53%
76%
89%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Firefox
Safari
Chrome
IE
NSS Labs Findings
• The browser is the first line of defense against multiple web-‐based threats; however, with a maximum historical protection rate of just 80 percent, the browser should not be the only line of defense.
• Products that do not provide the bulk of their protection in the earliest hours of an attack are not meeting the security requirements of today’s threatscape.
• Microsoft’s Internet Explorer continues to provide the best combination of malware and phishing protection.
• The application reputation technologies used by browsers from both Microsoft and Google provide a significantly safer browsing experience than do the browsers from Apple and Mozilla.
• User education is often better protection against social engineering attacks than browser technologies.
NSS Labs Recommendations
• Invest in awareness education about social engineering for all users.
• Evaluate trends that may indicate the need for browser replacement.
• Select and use security products that augment the protective capabilities of the browser.
Table of Contents
Overview ... 1
NSS Labs Findings ... 3
NSS Labs Recommendations ... 3
Analysis ... 6
Phishing Trends and Threats ... 6
NSS Empirical Results: Phishing Protection ... 7
NSS Empirical Results: Socially Engineered Malware Protection ... 9
Combined Protection Effectiveness ... 11
Aggregate Values ... 11
Weight A Minute ... 11
Evaluating the Data ... 11
The Great Equalizer ... 14
Appendix A: Raw Data ... 15
Appendix B: Weighting Formulas ... 17
Protection Multipliers: ... 17
Year Multipliers: ... 17
Reading List ... 19
Contact Information ... 20
Table of Figures
Figure 1 – Leading Browser Malware and Phishing Block Rates (2009, 2012, 2013) ... 1
Figure 2 – 2013 Combined Phishing And Malware Block Rates ... 2
Figure 3 – Unique Phishing Attacks ... 6
Figure 4 – APWG Phishing Uptime Statistics ... 7
Figure 5 – Mean Block Rate for Phishing ... 7
Figure 6 – Zero-‐Hour Phishing Block Rate ... 8
Figure 7 – Time to Block Phishing Attacks Relative to Uptime Trends (Hours) ... 9
Figure 8 – Mean Block Rate for Socially Engineered Malware ... 9
Figure 9 – Zero-‐Hour Socially Engineered Malware Block Rate ... 10
Figure 10 – Content Agnostic Malware Protection Breakout ... 10
Figure 11 – Combined Test Results (Not Weighted) ... 12
Figure 12 – Time Weighted ... 12
Figure 13 – Protection Weighted ... 13
Figure 14 – Time and Protection Weighted Scores ... 13
Figure 15 – 2013 Weighted Scores ... 13
Figure 16 – Mean Block Rate for Phishing ... 15
Figure 17 – Zero-‐Hour Block Rate ... 15
Figure 18 – Mean Block Rate for SEM ... 15
Figure 19 – Zero-‐Hour Block Rate for SEM ... 16
Figure 20 – Time Weighting ... 17
Figure 21 – Protection Type Weighting ... 17
Figure 22 – Type and Year Weighting ... 17
Analysis
Socially engineered malware and phishing attacks are two of the most significant threats against which web browsers must defend. NSS has for several years tested the leading browsers for their ability to protect against these attacks; however, each test has been presented as a stand-‐alone snapshot in time. While these real-‐world snapshot tests yield useful information, a correlated report is equally valuable in order to assess trends and establish vendor track records. This analyst brief examines the historical performance of browsers against phishing and against socially engineered malware attacks. The browsers are evaluated against each other and against the phishing threatscape. If the best performing product affords little protection, then the worst performing product is not significantly different. Fortunately, there are browsers that are addressing the challenges and that are able to provide significant protection for users.
Phishing Trends and Threats
The Anti-‐Phishing Working Group (APWG) has collected and published statistics about phishing attacks for several years. The APWG “Phishing Attack Trends”
1and “Global Phishing Survey”
2reports provide important insight into the phishing problem.
From Figure 3 it can be inferred that although the number of unique phishing emails and web sites has varied from 2009 to 2012, the scope of the problem remains significant. The uptick in unique phishing sites discovered in 2012 is not accompanied by a significant uptick in reports of unique phishing emails. While this may be due to under-‐
reporting of phishing email, it is likely indicative of cyber criminals increasingly using redirects in an attempt to compensate for the declining lifetimes of their attacks.
Figure 3 – Unique Phishing Attacks
1
http://www.apwg.org/resources/apwg-‐reports/
2
http://www.apwg.org/resources/apwg-‐reports/whitepapers
0 100,000 200,000 300,000 400,000 500,000 600,000 700,000
2009 2010 2011 2012
Unique Phishing Emails Reported Unique Phishing Web Sites Discovered
One of the critical metrics surrounding a browser’s effectiveness in combatting phishing attacks is how quickly it adds protection once an attack is live. Figure 4 illustrates the general decline in the lifetimes of phishing sites. In 2012, the average phishing site was live for just under 25 hours, and the median lifetime was approximately 12 hours. Products that do not provide the bulk of their protection in the earliest hours of an attack are not meeting the security requirements of today’s threatscape.
Figure 4 – APWG Phishing Uptime Statistics
NSS Empirical Results: Phishing Protection
NSS tested the leading browsers for phishing protection in 2009, 2012, and 2013, with the results presented in Figure 5 and Figure 6.
Figure 5 – Mean Block Rate for Phishing
In 2009, only IE and Firefox provided competitive block rates for phishing, with results in 2012 narrowing the differences to a four-‐point spread between browsers. In 2013, Firefox and Safari posted modest improvements in their scores, while Chrome dropped by 2 percent. IE has a trend of solid performance, but the browser’s phishing block rate declined from 92 percent in the 2012 “Browser Security Comparative Analysis Report” (CAR) on
“Phishing Protection” to 83 percent in the 2013 CAR on “Phishing Protection”.
0 20 40 60 80
1H2009 2H2009 1h2010 2h2010 1h2011 2h2011 1h2012 2h2012
Average Uphme (Hours) Median Uphme (Hours)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2012 2013
Chrome
Firefox
IE
Safari
Average
The lower results in the 2013 test should be evaluated against future tests to determine if the decline in IE’s mean block rate in the 2013 test indicates a problematic trend.
The time required to add new phishing sites is an important metric when determining the relevance of the mean block rate to consumer protection. For example, a browser that blocks more phishing sites in the first 12 hours will provide better protection than a less responsive browser that achieves a better block rate in the long run.
Historically, NSS testing has found that the browsers with the best early detection continue to lead until the end of the test; however, this may not always be the case.
Figure 6 – Zero-‐Hour Phishing Block Rate
During the 2013 test, all browsers showed improvement over their historical zero-‐hour block rates, as depicted in Figure 6. A metric that has a high correlation to the zero-‐hour block rate is the average time required to add protection for new phishing sites.
Figure 7 shows the APWG average phishing site uptime statistics and the mean phishing site uptime statistics overlaid with the results of the tested browser performance for average time to add protection for new phishing sites. (The APWG statistics for the first half of 2013 were not available at the time of writing.) Note that the browser phishing protection tests were performed only in 2009, 2012, and 2013, so performance in 2010 and 2012 is graphed in a linear fashion and may not reflect actual performance in those two years. Safari is the only browser to have had a worse response time to phishing attacks than either the mean or the average uptime for phishing sites since NSS began testing browsers. But Apple has dramatically improved its performance, and Safari posted the fastest response times in the most recent test report.
The median uptime for phishing attacks is significantly lower than the average response time and is the more important metric. All of the browsers are adding protection very quickly, with IE requiring 2.6 hours and Safari averaging 30 minutes. Firefox, however, has demonstrated the most consistent protection for phishing over time.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2012 2013
Chrome
Firefox
IE
Safari
Average
Figure 7 – Time to Block Phishing Attacks Relative to Uptime Trends (Hours)
NSS Empirical Results: Socially Engineered Malware Protection
Socially engineered malware (SEM) refers to an attack that deceives users into downloading and installing malicious software. In recent years, rogue antivirus programs have been at the forefront of SEM; however, there are many types of malicious programs that criminals use in conjunction with social engineering for financial gain.
Figure 8 demonstrates that there are, and historically have been, dramatic differences in browser protection against SEM.
Figure 8 – Mean Block Rate for Socially Engineered Malware
NSS conducted six browser malware protection tests between 2009 and 2013, and IE significantly outperformed the competition in all six texts. Only recently has Chrome become a viable option that provides significant malware protection for users. While Chrome, Firefox, and Safari all use Google’s Safe Browsing API, Chrome alone
incorporates Google’s Content Agnostic Malware Protection technology (CAMP). Prior to 2012, all three of the browsers using Google’s Safe Browsing API performed comparably.
0 10 20 30 40 50 60 70
2009 2010 2011 2012 2013
Chrome Firefox IE Safari
APWG Average Uphme APWG Median Uphme
u
Google Buys VirusTotal
0%
20%
40%
60%
80%
100%
2009 Q1 2010 Q3 2010 2011 2012 2013
Chrome Firefox IE Safari
When Google acquired VirusTotal, it was widely assumed that the service would be used to improve Chrome’s malware blocking abilities. The 2013 results do show an improvement in Chrome’s mean block rate, but also a significant drop in zero-‐hour protection. If Google continues to improve Chrome’s SEM protection, it may be difficult to differentiate the contribution of the VirusTotal acquisition from ongoing investment in Google’s Download Protection technology.
As with phishing attacks, response time is critical when providing protection against malware attacks. Figure 9 shows browser performance graphed at zero hour against SEM attacks. IE significantly outperforms the competition in all six tests. Both Firefox and Safari have declined in SEM protection since 2010.
Figure 9 – Zero-‐Hour Socially Engineered Malware Block Rate
Google’s Download Protection technology and Microsoft’s App Rep technologies are the reason that Chrome and IE are able to block such high percentages of SEM. Neither Chrome nor IE relies on the certain knowledge that a file is bad; rather, they block files that do not meet reputational criteria.
Figure 10 – Content Agnostic Malware Protection Breakout
Figure 10 displays the combination of SEM-‐blocking technologies used by Chrome and IE. Both URL and CAMP protection methodologies can suffer from false positives; however, the more important consideration is the protection that URL reputation adds over CAMP.
u
