1
Banker Malware Protection Test Report
A test commissioned by Kaspersky Lab and performed by AV-Test GmbHDate of the report: April 10th, 2012, last update: April 10th, 2012
Executive Summary
In March 2012, AV-Test performed a comparative review of 3 home user security products to determine their capabilities to protect against malware that is related to Online Banking. Three individual tests have been performed. The first was a dynamic-detection test of new malware with 45 samples, the second was a remediation test with 25 samples and the final test was a static detection test with 1867 samples. To perform the test runs, a clean Windows 7 image was used on several identical PCs. On this image, the security software was installed and then the individual tests have been carried out. In case of the dynamic test, the samples have been executed and any detection by the security software was noted. Additionally the resulting state of the system was compared with the original state before the test in order to determine whether the attack was successfully blocked or not. For the remediation test the computer has been infected and then the security software had to clean the infection. The final state of the system has been noted in order to determine the
cleaning success. In case of the static detection test, the products had to scan a new set of files every day over a period of twelve days. Detections have been noted to determine the cumulated daily detection result. Additionally a final scan over all files has been carried out to determine the detection rate seven days after the last scan.
The products were pretty close in most parts of the tests which shows that big efforts are made to protect against Online Banking malware. The overall best result in the described test was achieved by the Kaspersky product.
2
Overview
With the increasing number of threats that is being released and spreading through the Internet these days, the danger of getting infected is increasing as well. A few years back there were new viruses released every few days. This has grown to several thousand new threats per hour.
Figure 1: New samples added per year
In the year 2000, AV-Test received more than 170,000 new samples, and in 2010 and 2011, the number of new samples grew to nearly 20,000,000 new samples each. The numbers continue to grow in the year 2012 with already over 5 million new samples in the first quarter. The growth of these numbers is displayed in Figure 1.
Since the main motivation of malware these days is financial gain, this report focuses on malware that is related to online banking or similar credential stealing attacks. We covered dynamic detection, static detection and remediation of these threats found in February 2012 and March 2012.
Products Tested
The testing occurred in February and March 2012. AV-Test used the latest releases available at the time of the test of the following products:
Bitdefender Internet Security 2012 Kaspersky Internet Security 2012 Norton Internet Security 2012
Methodology and Scoring
Platform
All tests have been performed on identical PCs equipped with the following hardware: Intel Xeon Quad-Core X3360 CPU
0 2 000 000 4 000 000 6 000 000 8 000 000 10 000 000 12 000 000 14 000 000 16 000 000 18 000 000 2005 2006 2007 2008 2009 2010 2011* 2012*
New unique samples added to AV-Test's malware repository
(2005-2012)
Dec Nov Oct Sep Aug Jul Jun May Apr Mar3 4 GB Ram
500 GB HDD (Western Digital)
Intel Pro/1000 PL (Gigabit Ethernet) NIC
The operating system was Windows 7 Ultimate Service Pack 1 with only those hotfixes that were part of SP1 as well as all patches that were available on February 1st 2012.
Testing methodology General
1. Clean system for each sample. The test systems should be restored to a clean state before being exposed to each malware sample.
2. Physical Machines. The test systems used should be actual physical machines. No Virtual Machines should be used.
3. Product Cloud/Internet Connection. The Internet should be available to all tested products that use the cloud as part of their protection strategy.
4. Product Configuration. All products were run with their default, out-of-the-box configuration.
5. Sample Cloud/Internet Accessibility. If the malware uses the cloud/Internet connection to reach other sites in order to download other files and infect the system, care should be taken to make sure that the cloud access is available to the malware sample in a safe way such that the testing network is not under the threat of getting infected.
6. Allow time for sample to run. Each sample should be allowed to run on the target system for 10 minutes to exhibit autonomous malicious behavior. This may include initiating
connections to systems on the internet, or installing itself to survive a reboot (as may be the case with certain key-logging Trojans that only activate fully when the victim is performing a certain task).
The procedures below are carried out on all tested programs and all test cases at the same time in order to ensure that all protection programs have the exact same test conditions. If a test case is no longer working or its behavior varies in different protection programs (which can be clearly
determined using the Sunshine analyses), the test case is deleted. This ensures that all products were tested in the exact same test scenarios. All test cases are solely obtained from internal AV-TEST sources and are always fully analyzed by AV-TEST. We never resort to using test cases or analyses provided by manufacturers or other external sources.
Dynamic Test
1. The products are installed, updated and started up using standard/default settings. The protection program has complete Internet access at all times.
2. AV-TEST uses the analysis program Sunshine, which it developed itself, to produce a map of the non-infected system.
3. It then attempts to execute the malicious file.
4. If execution of the sample is blocked with static or dynamic detection mechanisms by the program, this is documented.
5. If execution is not blocked, an on-demand scan (Full computer scan) will be carried out. Any detections will be noted and any removal actions will be allowed.
4 6. Given that the detection of malicious components or actions is not always synonymous to
successful blockage, Sunshine constantly monitors all actions on the computer in order to determine whether the attack was completely or partially blocked or not blocked at all. 7. A result for the test case is then determined based on the documented detection according
to the protection program and the actions on the system recorded by Sunshine.
Remediation Test
1. The products are installed, updated and started up using standard/default settings. The protection program has complete Internet access at all times.
2. AV-TEST uses the analysis program Sunshine, which it developed itself, to produce a map of the non-infected system.
3. The guard of the product is disabled and configured to be enabled again after a reboot. The malware is executed to infect the system and a reboot is initiated.
4. After the system comes back up, any detections of the guard are noted and removal actions will be allowed.
5. Additionally, an on-demand scan (Full computer scan) will be carried out. Any detections will be noted and any removal actions will be allowed.
6. If there any instructions by the program these will be followed.
7. A final reboot is performed and the final system state is stored with a Sunshine snapshot 8. Given that the detection of malicious components or actions is not always synonymous to
successful removal, Sunshine constantly monitors all actions on the computer in order to determine whether the attack was completely or partially removed or not removed at all. 9. A result for the test case is then determined based on the documented detection according
to the protection program and the actions on the system recorded by Sunshine.
Static Scanning Test
1. The test ran in the timeframe from February 23rd to March 8th, 2012
2. Every working day new malware samples have been scanned with the products with recent updates. So there were twelve scans in total.
3. Finally a rescan of all scanned samples has been performed on March 16th, 2012
4. All samples that are related to online banking have then been selected to calculate the final result, two values are given:
a. The cumulated daily detection rate
b. The detection rate of the whole set with a rescan on March 16th, 2012
Samples
The malware set for the dynamic test contains 45 samples related to online banking. These files have been collected during March 5th and March 14h 2012. The malware set for the remediation test contains 25 samples related to online banking. These files have been collected during February 24th and March 9h 2012. The malware set for the static scanning test contains 1867 samples related to online banking. These files have been collected during February 23rd and March 8th 2012. Every sample has been tested on the day of its appearance in AV-TESTs analysis systems.
5
Test Results
The test results clearly show that all three products do handle malware related to online banking very well. There are a few slight differences between the three vendors, partially due to their approach to protecting the computer.
One of these differences becomes obvious when looking at the first category, the static detection of malware which is shown in Figure 2.
Figure 2: Static Detection of Malware
All products detected over 90% of the malware with the rescan on March 16th. Norton is somewhat behind the other two products though. The same is true for the cumulated result of the daily scan. While Bitdefender and Kaspersky show very good detection rates, Norton is clearly behind. The reason for that is their approach to detection. This test was looking at the static file scanner detection of inactive files. However, Norton makes heavy use of their online reputation service (Norton Insight) to detect new files that are downloaded or executed, which was not part of this test. This explains why the cumulated result of the daily scan is so low. Most of the missing 60% would probably be detected with that (or another) approach. This becomes obvious when looking at the next test, the dynamic detection of new malware, where Norton shows much higher detection rates. Still, this test shows that products do still heavily rely on static detection. Even a product like Norton adds most of the new malware to their signature database sooner or later, which can be seen by the results of the rescan.
The next test is the dynamic detection of new malware which result is shown in Figure 3. All three products had excellent detection rates, with over 95% of the tested malware samples. The best result comes from Kaspersky, very closely followed by Bitdefender and Norton. This test enables the
products to use their more advanced detection features, such as online reputation and behavioral
94,6% 99,3% 38,7% 99,5% 99,6% 90,6% 0,0% 10,0% 20,0% 30,0% 40,0% 50,0% 60,0% 70,0% 80,0% 90,0% 100,0%
Bitdefender Internet Security 2012
Kaspersky Internet Security 2012
Norton Internet Security 2012
Static Detection of Malware
6 detection. As mentioned before, this is what Norton primarily uses to detect new threats and the results show that this is working very well, resulting in better numbers than the static detection test.
Figure 3: Dynamic Detection of Malware
This test also shows, that the good static detection rates of Bitdefender and Kaspersky do help in the dynamic detection as well and sets them a bit ahead of Norton. This is because new malware often downloads or drops older components that are already covered with signatures. So even if the new sample is not detected by signatures, there is still a good chance that the additional components will be detected by them. The key is to combine different detection techniques to get the best coverage of new malware.
Even if a product combines all the different approaches, it will never protect 100% against all threats. So there is always the chance of infecting a system. While there is consensus in IT-Security that reimaging the system and using backup data after a malware infection is the preferred approach, this is not always feasible. Especially in the home user domain the reality is that backups are usually not done at all or only once in a while, being too old to be useful. Also many home users don’t have the technical expertise to perform a reinstallation of their system or simply don’t want to spend the time. So they are looking for more convenient ways to get rid of the infection and this is where the system disinfection capabilities of security software become important. The results of this test are shown in Figure 4. Please note that no additional remediation tools or rescue media have been used to perform this test. So the results show the capabilities of the product itself only.
Kaspersky detected the most malware threats (24 out of 25) and also fully disabled them, however leaving a few traces of other malware infections behind. Bitdefender detected one malware sample less than Kaspersky, but was able to perfectly clean all of the detected ones (23 of 25). Norton missed to detect 4 active threats and had a bit more trouble in cleaning them.
97,8% 95,6% 100,0% 100,0% 95,6% 91,1% 95,6% 97,8% 86,7% 0,0% 20,0% 40,0% 60,0% 80,0% 100,0%
Bitdefender Internet Security 2012
Kaspersky Internet Security 2012
Norton Internet Security 2012
Dynamic Detection of Malware
Overall Detection (Warning) Rate Overall Detection and Blocking Rate
7
Figure 4: System Disinfection/Remediation
There are a few interesting findings from these results. The first is that no product was able to detect and disable all of the infections. Usually this is not because there are no detection routines for that threat but because the threat actively tries to fight the security product by disabling or even removing it from the system. This explains why additional cleaning tools and rescue media are sometimes important for successful cleaning. Especially Norton seems to rely a lot on those, which is not a big surprise. This product is one of the most targeted by malware, so it is more difficult for them to clean with the product only and they have to rely on additional cleaning tools. Furthermore this shows the importance of the backup and reinstalling/reimaging approach. Even though cleaning (especially when it comes to the active components of malware) works very well for the three tested products, none of it was able to fully detect and disable all of the threats. So there is always a small uncertainty factor. 92% 96% 84% 92% 96% 76% 92% 84% 32% 0% 20% 40% 60% 80% 100%
Bitdefender Internet Security 2012
Kaspersky Internet Security 2012 Norton Internet Security 2012
System Disinfection/Remediation
Detection rate (detected files and/or Registry values) Disinfection rate (all active components)
8
Appendix
Version information of the tested software
Developer, Distributor Product name Program version Engine/ signature version
Bitdefender Bitdefender Internet Security 2012 15.0.36.1530 7.41.109
Kaspersky Lab Kaspersky Internet Security 2012 12.0.0.374 (h) 16.5.0.1
Symantec Norton Internet Security 2012 19.5.0.145 n/a
Copyright © 2012 by AV-Test GmbH, Klewitzstr. 7, 39112 Magdeburg, Germany Phone +49 (0) 391 60754-60, Fax +49 (0) 391 60754-69, Web http://www.av-test.org
