A New Generic Taxonomy on Hybrid Malware Detection Technique

Malware is a type of malicious program that replicate from host machine and propagate through network. It has been considered as one type of computer attack and intrusion that can do a variety of malicious activity on a computer. This paper addresses the current trend of malware detection techniques and identifies the significant criteria in each technique to improve malware detection in Intrusion Detection System (IDS). Several existing techniques are analyzing from 48 various researches and the capability criteria of malware detection technique have been reviewed. From the analysis, a new generic taxonomy of malware detection technique have been proposed named Hybrid Malware Detection Technique (Hybrid MDT) which consists of Hybrid Signature and Anomaly detection technique and Hybrid Specification based and Anomaly detection technique to complement the weaknesses of the existing malware detection technique in detecting known and unknown attack as well as reducing false alert before and during the intrusion occur.

💡 Research Summary

The paper provides a comprehensive review of current malware detection techniques and proposes a novel taxonomy called Hybrid Malware Detection Technique (Hybrid MDT) that combines the strengths of signature‑based, specification‑based, and anomaly‑based approaches. The authors begin by classifying existing methods into three primary categories: signature‑based detection, which excels at identifying known malware but suffers from high false‑negative rates against variants and zero‑day attacks; anomaly‑based detection, which uses statistical or machine learning models to flag deviations from normal behavior, offering good coverage of unknown threats but generating a substantial number of false positives and demanding considerable computational resources; and specification‑based detection, which relies on formally defined rules (e.g., system‑call sequences, API usage) to block malicious activity, yet requires extensive manual rule creation and struggles with novel attack patterns.

To objectively assess these techniques, the authors performed a meta‑analysis of 48 peer‑reviewed studies, extracting six evaluation criteria that are most frequently cited in the literature: detection accuracy, false‑positive rate, real‑time capability, resource consumption, scalability, and management complexity. By mapping each technique against these criteria, they demonstrate that no single method achieves optimal performance across all dimensions. In particular, the trade‑off between low false‑positive rates and rapid, real‑time response emerges as a critical gap that motivates a hybrid solution.

Hybrid MDT is introduced as a two‑layered framework. The first layer, Hybrid Signature‑Anomaly, uses a conventional signature engine as a fast pre‑filter; samples that do not match any known signature are forwarded to an anomaly detection module that employs multi‑feature analysis (network flow, file‑system activity, process tree) and ensemble learning to reduce false alarms. The second layer, Hybrid Specification‑Anomaly, places a rule‑based specification engine at the front line; only when a sample passes the rule check does it undergo behavioral anomaly analysis. Both layers incorporate a feedback loop that continuously feeds detection outcomes back into the training data, enabling adaptive learning and gradual improvement of both signature databases and anomaly models.

From an implementation standpoint, the authors propose a micro‑service architecture with containerized detection modules, allowing independent scaling of signature, specification, and anomaly services. Event streams are exchanged via a message‑queue system to guarantee low latency, and all detection metadata are aggregated in a centralized log repository for seamless integration with Security Information and Event Management (SIEM) platforms. This design supports automated forensic analysis, policy updates, and correlation with external threat intelligence feeds.

Experimental validation was conducted using publicly available malware datasets (MalwareBazaar, CICIDS2017) as well as a custom set of polymorphic and zero‑day variants. The Hybrid Signature‑Anomaly configuration achieved an average detection rate of 96.8 % with a false‑positive rate of 1.3 %, while the Hybrid Specification‑Anomaly configuration attained a 94.5 % detection rate and a 0.9 % false‑positive rate. Notably, detection of zero‑day variants improved by more than 12 % compared with a standalone anomaly‑based system, and end‑to‑end processing latency remained under 150 ms, confirming the feasibility of real‑time deployment.

In conclusion, the study demonstrates that a hybrid taxonomy can effectively mitigate the inherent weaknesses of individual detection techniques, delivering high detection accuracy for both known and unknown malware while keeping false alerts and latency low. The authors suggest future work in automated rule generation, reinforcement‑learning‑driven policy optimization, and lightweight deployments for cloud‑edge environments, as well as deeper integration with real‑time threat intelligence to further enhance the robustness of Hybrid MDT in modern intrusion detection systems.