ATP test for stealthy malware lurking on Windows systems

Each security solution in the test needed to provide protection against 5 samples of ransomware and 5 samples of infostealers. There are different steps for these two malware categories when it comes to protection. For this reason, a product can earn 3 points per ransomware it fends off and 4 points per infostealer. If an attack is only partially thwarted in one step, half points are awarded accordingly. Overall, the products are able to earn up to 35 points for their protection score in the table. The 10 malware samples employed the following techniques in their attack, either individually or in concert:

Backdoored binary: In this type of attack, if a legitimate executable file is not yet signed, attackers add malicious code to it. This conceals the malicious operations as they come from a process that could actually have been executed by a normal user.

In our testing, we injected the malware into a legitimate executable 7zFM.exe file of the free, widely available software 7-Zip, which is distributed without a cryptographic signature. Very often the reason behind is the associated costs and organizational effort for receiving a valid code signing certificate. Smaller not-for-profit and open source projects will often not sign their EXE and DLL files or they will not sign them regularly.

Heaven’s Gate: This is a technique helping malware mask its behavior pattern from detection tools. It exploits a feature in the operating system in order to enable a malicious 64-bit code to run within a 32-bit process. Using this compatibility layer directly makes it more difficult for security software to track operations because switching to 64-bit code inside a 32-bit process may not be expected and, in turn, is ignored.

In our testing, we use Heaven’s Gate for process injection and run our 64-bit malware in a legitimate Windows 32-bit process.

Every attack carried out in the test starts in the same way: A spear phishing e-mail arrives on a Windows system and includes an archive in its payload. The 10 scenarios involve either an executable EXE file, an MSI installation file, or a 7-zip executable file. All variants contain malicious code disguised using the various techniques. The respective detection, defense and protection steps are described in greater in the individual scenarios.

The 10 test scenarios

All attack scenarios are documented according to the standard of the MITRE ATT&CK database. The individual sub-techniques are listed in the MITRE database for "Techniques", for example "T1566.001" under "Phishing: Spearphishing Attachment”. Each test step is thus defined among the experts and can be logically understood. In addition, all attack techniques are explained, along with how the malware infection occurs and impacts systems. 

The ATP test: here’s how well the products for consumer users performed

The vendors of the 9 solutions for consumer users evaluated in the test have reason to celebrate. Why? Because 8 products passed the ATP test without a glitch, thereby scoring the full 35 points for their protection level. These were the solutions from Avast, AVG, Bitdefender, ESET, Kaspersky, McAfee, Microsoft and Norton.

It was only Avira Security for Windows that experienced issues with one ransomware and one infostealer attack. The malware was detected by the solution, but it was unable to block it completely, meaning that the attack could get started, but the attackers were not able to proceed very far. Additional protection mechanisms eventually took effect and fended off the attack. Notwithstanding, the delayed response cost Avira a one-point deduction two times. This left the solution with 33 out of 35 points in terms of the protection score.

All tested packages for consumer users were awarded the AV-TEST "Advanced Certified" certificate, as they achieved at least 75 percent of the maximum 35 points (26.5 points) in the test.

The ATP test: here’s how well the products for corporate users performed

The outcome of the solutions for corporate users also looks very promising. 10 of the 12 products tested completed the ATP test with the best score of 35 points in terms of protection – a perfect performance for the products from Avast, Bitdefender, Huawei, Kaspersky (both versions), Microsoft, Microworld, Symantec, Trellix and WithSecure.

There were only a few minor issues with the solutions from ESET and Qualys. For example, ESET detected all 10 attackers, but it could not completely block one infostealer. The malware was able to proceed with its attack, but it was then blocked and subsequently dealt with by the downstream protective mechanisms. Nevertheless, this cost ESET a one-point deduction, leading to a final score of 34 out of 35 points.

The issue that Qualys encountered with an infostealer is unfortunately a bit more wide-reaching. The solution did manage to identify the attacker; however, it was not able to stop it. Even downstream protection mechanisms only prevented parts of the attack. But in the end the infostealer gained the upper hand and stole the data, which cost Qualys crucial points, earning only 32.5 out of 35 points for overall protection.

All the tested solutions for corporate users received “Advanced Approved Endpoint Protection” certification, as they achieved the required 75 percent (at least 26.5 points) out of 35 points for the protection score.

Conclusion: Many high fliers in the ATP test

This ATP test took a closer look at 21 protection solutions. Of these, 18 demonstrate perfect performance in defending against the latest techniques used by infostealers and ransomware. They all deserve the full 35 points as proof of their excellent level of protection.

When it comes to the 9 packages for consumer users, it was only Avira that missed out on a few points due to minor errors. Nevertheless, the outcome is certainly impressive. The other 8 protection packages do their jobs without fail.

Of the 12 solutions for corporate users, 10 produced completely error-free results, earning them the full set of points. The product from ESET managed to eventually correct the minor error it made and in the end protected the Windows system. Only Qualys lost data to an attacker in one case, but this means it also lost crucial points.