ATP TEST: 26 security packages fend off ransomware and info stealers

All the products had to successfully defend against 5 ransomware variants and 5 info stealers in 10 real-life scenarios. Each routine in an attack is documented in all its steps and later evaluated. Each attack is described in a matrix according to the MITRE ATT&CK standard. In terms of evaluation, the lab awards points for each detected and thwarted attack. The test involved 5 ransomware samples and 5 info stealer. With ransomware, there are three key steps to recognize, with info stealers there are four actions. A product can earn a half or full point for each action that is fended off. The lab is able to award up to 3 points five times for each thwarted ransomware sample, and 4 points five times for the info stealers. If the attacker is only stopped partially in a defensive step, this accordingly results in only half a point or none at all. The highest value a product is able to achieve in the protection score is 35 points.

The attacks involve various attack techniques currently used by attackers in real-life assaults. Cyberattackers partly also combine various techniques to make detection more difficult. Here are two attack techniques that saw extremely heavy use in this test.

Memory-mapped file I/O: The technique is an alternate method for accessing data on a drive. File access is handled via memory, whereby memory-related addressing is used. For example, ransomware can encrypt cached documents without causing additional write operations on the hard drive. Disk activity monitoring tools will not notice that the malware is accessing a cached document because the data is being served from memory and not from the disk drive.

In our attacks, memory-mapped file I/O is used to encrypt files with ransomware or to read files during exfiltration.

Command line and PPID spoofing: Command line spoofing and PPID (Parent Process ID) spoofing are techniques used by malware to infiltrate legitimate system activities. When obfuscated in this way, they seek to evade detection. Command line spoofing disguises the arguments passed to malicious processes. PPID spoofing involves manipulating the parent process ID of a malicious process to make it appear that it was launched by a trusted process. These techniques complicate forensic analysis and hinder detection of malicious activity by security tools that analyze process relationships or command line activity.

In our attacks, PowerShell instances are launched to perform isolated tasks. These instances are executed with fake PPIDs and command lines.

The 10 test scenarios

All attack scenarios are documented according to the standard of the MITRE ATT&CK database. The individual sub-techniques are listed in the MITRE database for “Techniques”, for example “T1566.001” under “Phishing: Spearphishing Attachment“. Each test step is thus defined among the experts and can be logically understood. In addition, all attack techniques are explained, along with how successful the malware is.

For consumer users: the ATP test results

The test of products for consumer users yielded some surprises. Only 4 out of the 11 products tested completed the test perfectly in all 10 scenarios and received 35 points for the protection score: Bitdefender, ESET, McAfee and Microsoft.

Although the packages from Avast, AVG and Norton detected the attackers in all 10 scenarios, they grappled with an info stealer and with a ransomware: The products identified the attackers, but were unable to block them completely. Both malware samples used a technique for their attacks in which an AutoHotKey script was created and used. The products could not prevent this from being created either. It did work, however, once the script was executed further – stopping the attack. This resulted in each product losing 1 point per malware problem and thus 2 points in total. In the end, Avast, AVG and Norton each earned 33 points for the protection score.

G DATA and F-Secure each detected only 9 of the attackers. Both failed against another data stealer, losing 4 points as a result. In addition, each package recognized a ransomware sample but did not block it in the initial steps. The two solutions were also unable to prevent the generation of an attack script. During its execution, at least the G DATA package stopped the attack and thus only lost one point. The F-Secure security package was not capable of this, and the system was encrypted. This resulted in an additional 2.5 points lost. At the end of the test, G DATA scored 30 out of 35 points and F-Secure 28.5 points for the protection score.

The situation with Avira and K7 Computing can be quickly summed up: They identified only 8 out of 10 attacks in the test. With one ransomware and one info stealer each, they put up no defense, thus losing a full 7 points (4 and 3 points). Both ended up with 28 instead of 35 points in the protection score.

All protection packages earned the "Advanced Certified" certificate from AV-TEST, as they achieved 75 percent of the maximum 35 points (26.5 points).

For corporate users: the ATP test results

The 15 evaluated solutions for companies in the ATP test exhibited strong performance. Attackers didn't stand a chance against 12 of the 15 products in all 10 scenarios. For this, the following products received the full 35 points for the protection score: Acronis, Bitdefender (both versions), ESET, Kaspersky (both versions), Microsoft, Microworld, Qualys, Symantec, Trellix and WithSecure.

The Net Protector solution experienced minor difficulties in detecting a ransomware: the attacker was detected but not completely blocked. The ransomware was still able to load an attack script. Upon being executed, however, the attack was stopped. This cost a one-point deduction, with a final core of 34 out of 35 points.

Avast's endpoint solution also struggled with an info stealer and a ransomware. The attacker was detected, but was initially able to continue working unhindered. The malware even created the attack scripts in both cases. Once they were launched, however, Avast was successful in completely stopping the attackers. Due to the problems, the product had a point deducted twice, and ended up with 33 out of 35 points on the protection score.

Rapid7's protection solution detected all 10 attackers in the test, but hit its limits with two of them. With one info stealer, it was a to-and-fro battle: The attacker was recognized initially, but could not be stopped by the solution. A script was then created which began collecting the data. Rapid7 was not able to prevent its execution. In the end, the malware even managed to extract and send off some data. Only then did the protection software succeed in stopping the attacker. This first incident cost Rapid7 2 out of 3 points.

The second complication, this time involving ransomware, unfolded almost identically. However, following initial detection, Rapid7 was no longer able to have any impact on the attacker. In the end, the system was encrypted. That is why the Rapid 7 solution achieved 29.5 out of 35 points for the protection score.

All corporate user products in the ATP test received “Advanced Approved Endpoint Protection” certification, as they achieved 75 percent (at least 26.5 points) out of 35 points for the protection score.

Some products need to improve

With its detailed trials, the ATP test reveals that there are both very good protection packages and some that still have room for improvement, particularly in the case of products for consumer users. Only 4 of the 11 system protectors examined earned the maximum number of points in the test: Bitdefender, ESET, McAfee and Microsoft.

The security suites from Avast, AVG and Norton still delivered strong performance with 33 out of 35 points each. Although they were initially losing against the malware, they regained full control of the situation.

The situation looks much better among solutions for corporate users. 12 of the 15 tested endpoint packages for Windows passed the test without any errors, crossing the finish line with the full 35 points. Only 3 vendors need to improve.