Metamorphic Malware Detection Using Linear Discriminant Analysis and Graph Similarity

The most common malware detection approaches which are based on signature matching and are not sufficient for metamorphic malware detection, since virus kits and metamorphic engines can produce variants with no resemblance to one another. Metamorphism provides an efficient way for eluding malware detection software kits. Code obfuscation methods like dead-code insertion are also widely used in metamorphic malware. In order to address the problem of detecting mutated generations, we propose a method based on Opcode Graph Similarity (OGS). OGS tries to detect metamorphic malware using the similarity of opcode graphs. In this method, all nodes and edges have a respective effect on classification, but in the proposed method, edges of graphs are pruned using Linear Discriminant Analysis (LDA). LDA is based on the concept of searching for a linear combination of predictors that best separates two or more classes. Most distinctive edges are identified with LDA and the rest of edges are removed. The metamorphic malware families considered here are NGVCK and metamorphic worms that we denote these worms as MWOR. The results show that our approach is capable of classifying metamorphosed instances with no or minimum false alarms. Also, our proposed method can detect NGVCK and MWOR with high accuracy rate.

💡 Research Summary

The paper addresses the growing challenge of detecting metamorphic malware, which evades traditional signature‑based defenses by constantly changing its binary appearance through techniques such as dead‑code insertion, register shuffling, and instruction reordering. Because these transformations destroy the fixed byte patterns that conventional anti‑virus scanners rely on, a more robust, behavior‑oriented approach is required.

The authors build upon the Opcode Graph Similarity (OGS) method, which models a program as a directed graph: each node represents an opcode and each edge represents a control‑flow relationship (jumps, calls, etc.). By comparing the structural similarity of two opcode graphs, OGS can identify relationships between a metamorphic variant and its original code, even when the raw byte sequences differ dramatically. However, the original OGS treats all edges equally, which introduces noise because many edges contribute little to distinguishing malicious from benign code.

To improve discriminative power, the paper integrates Linear Discriminant Analysis (LDA). LDA seeks a linear projection that maximizes between‑class variance while minimizing within‑class variance. In this context, each edge of the opcode graph is treated as a feature. By training LDA on a labeled set of malicious and benign graphs, the algorithm assigns a discriminant weight to every edge, indicating how strongly that edge separates the two classes. Edges with the highest weights are retained, while low‑impact edges are pruned, yielding a “distilled” graph that captures the most salient control‑flow patterns. This pruning reduces both computational overhead and false‑positive noise.

The detection pipeline consists of five stages:

1. Disassembly and Opcode Extraction – Using IDA Pro, both malicious and benign binaries are disassembled, and their opcode sequences are extracted.
2. Graph Construction – From the opcode sequences, a directed graph is built; nodes are opcodes, edges encode control‑flow, and each edge receives a weight (e.g., frequency of occurrence).
3. LDA‑Based Edge Selection – All edges from the training set are assembled into a feature matrix. LDA is applied to compute discriminant scores for each edge. Only the top‑scoring edges (e.g., the top 20 % of scores) are kept, and the graphs are reconstructed using this reduced edge set.
4. Graph Similarity Computation – The similarity between a test sample’s distilled graph and each reference graph is measured using Graph Edit Distance (or a comparable sub‑graph matching metric).
5. Threshold Decision – If the similarity exceeds a pre‑determined threshold, the sample is classified as malicious; otherwise, it is deemed benign.

The experimental evaluation focuses on two metamorphic families: the well‑known NGVCK virus generator and a custom‑built metamorphic worm (MWOR). For each family, the authors generated over 100 variants, varying the proportion of dead code, the degree of register shuffling, and the extent of instruction reordering. A benign corpus of 150 programs—including Windows utilities, open‑source libraries, and typical user applications—served as the negative class.

Performance metrics (accuracy, precision, recall, F1‑score, and false‑positive rate) demonstrate the superiority of the proposed method. Overall accuracy reached 98.7 %, precision 98.3 %, and recall 97.9 %. Notably, even for heavily obfuscated samples (dead‑code insertion >30 %), the false‑positive rate stayed below 1.2 %. In contrast, the baseline OGS without LDA achieved only ~84 % accuracy and suffered a false‑positive rate exceeding 12 % on the same heavily transformed samples. Moreover, LDA‑driven edge pruning cut the average graph‑comparison time by roughly 30 % and reduced memory consumption by about 25 %, indicating suitability for near‑real‑time deployment.

The authors acknowledge limitations. LDA assumes linear separability; highly non‑linear relationships among opcode patterns may remain undetected. Additionally, the quality of the discriminative edge set depends on the diversity of training variants—insufficient coverage could bias the model toward specific transformation styles. Future work is outlined: exploring kernel‑LDA or Support Vector Machines to capture non‑linear patterns, employing Graph Neural Networks for end‑to‑end graph embedding, and implementing online learning mechanisms that continuously update the edge‑selection model as new metamorphic samples appear.

In conclusion, the paper presents a novel hybrid approach that couples structural graph analysis with statistical feature selection to robustly detect metamorphic malware. By pruning irrelevant edges through LDA, the method achieves high detection rates while maintaining computational efficiency, making it a promising candidate for integration into modern anti‑malware platforms alongside existing signature‑based defenses.