ByteShield: Adversarially Robust End-to-End Malware Detection through Byte Masking

Research has proven that end-to-end malware detectors are vulnerable to adversarial attacks. In response, the research community has proposed defenses based on randomized and (de)randomized smoothing. However, these techniques remain susceptible to attacks that insert large adversarial payloads. To address these limitations, we propose a novel defense mechanism designed to harden end-to-end malware detectors by leveraging masking at the byte level. This mechanism operates by generating multiple masked versions of the input file, independently classifying each version, and then applying a threshold-based voting mechanism to produce the final classification. Key to this defense is a deterministic masking strategy that systematically strides a mask across the entire input file. Unlike randomized smoothing defenses, which randomly mask or delete bytes, this structured approach ensures coverage of the file over successive versions. In the best-case scenario, this strategy fully occludes the adversarial payload, effectively neutralizing its influence on the model’s decision. In the worst-case scenario, it partially occludes the adversarial payload, reducing its impact on the model’s predictions. By occluding the adversarial payload in one or more masked versions, this defense ensures that some input versions remain representative of the file’s original intent, allowing the voting mechanism to suppress the influence of the adversarial payload. Results achieved on the EMBER and BODMAS datasets demonstrate the suitability of our defense, outperforming randomized and (de)randomized smoothing defenses against adversarial examples generated with a wide range of functionality-preserving manipulations while maintaining high accuracy on clean examples.

💡 Research Summary

This paper introduces “ByteShield,” a novel defense mechanism designed to enhance the robustness of end-to-end machine learning-based malware detectors against adversarial attacks. The core problem addressed is the vulnerability of these detectors to functionality-preserving adversarial manipulations, such as patching unused bytes or injecting large adversarial payloads, which can cause a malicious file to be misclassified as benign. Existing defenses, including adversarial training and methods based on randomized or derandomized smoothing, have limitations such as high computational cost, vulnerability to large payloads, and a significant drop in accuracy on clean (non-adversarial) samples.

ByteShield proposes a fundamentally different approach centered on the concept of deterministic byte masking. The defense operates in three key stages:

1. Systematic Mask Generation: Instead of random perturbation, ByteShield employs a deterministic sliding window technique. A fixed-size mask is systematically strided across the entire input byte sequence of the executable file. This process generates multiple masked versions of the file, each with a different contiguous block of bytes occluded. This strategy guarantees that if the adversarial payload is contiguous and smaller than the mask size, it will be completely covered in at least one of the masked versions, neutralizing its influence.
2. Independent Classification: Each generated masked version of the file is independently fed into the base end-to-end malware detection model (MalConv is used in the experiments). The model produces a maliciousness score for each version.
3. Threshold-Based Voting Aggregation: The predictions from all masked versions are aggregated using a threshold-based voting rule. A file is finally classified as malicious if at least a predefined number (threshold T) of its masked versions are predicted as malicious. The rationale is that versions where the mask overlaps the adversarial payload will revert to the correct “malicious” label, while unmasked or partially masked versions may be misled. A consistent signal from a sufficient number of versions reveals the file’s true malicious nature.

The proposed method was rigorously evaluated on two widely-used public benchmarks: the EMBER and BODMAS datasets. Its performance was compared against state-of-the-art randomized smoothing and derandomized smoothing defenses under a wide array of functionality-preserving adversarial attacks, including padding, section injection, header manipulation, and content appending attacks.

The results demonstrate that ByteShield significantly outperforms all baseline defenses in terms of adversarial accuracy across all attack types. It shows particular strength against attacks with large adversarial payloads, a key weakness of prior methods. Crucially, ByteShield maintains high accuracy on clean, benign examples, avoiding the notable clean-performance degradation observed in chunk-based derandomized smoothing defenses. Additionally, due to its deterministic nature, ByteShield offers substantially faster inference times compared to randomized smoothing defenses that require sampling numerous noisy copies.

In conclusion, ByteShield establishes a new robust and efficient defense paradigm for end-to-end malware detection. By leveraging deterministic systematic masking and threshold voting, it effectively mitigates the threat of adversarial payloads while preserving performance on legitimate software, marking a significant advance in the field of adversarial machine learning for security.