Acceleration of Statistical Detection of Zero-day Malware in the Memory Dump Using CUDA-enabled GPU Hardware

This paper focuses on the anticipatory enhancement of methods of detecting stealth software. Cyber security detection tools are insufficiently powerful to reveal the most recent cyber-attacks which use malware. In this paper, we will present first an idea of the highest stealth malware, as this is the most complicated scenario for detection because it combines both existing anti-forensic techniques together with their potential improvements. Second, we present new detection methods, which are resilient to this hidden prototype. To help solve this detection challenge, we have analyzed Windows memory content using a new method of Shannon Entropy calculation; methods of digital photogrammetry; the Zipf Mandelbrot law, as well as by disassembling the memory content and analyzing the output. Finally, we present an idea and architecture of the software tool, which uses CUDA enabled GPU hardware to speed-up memory forensics. All three ideas are currently a work in progress.

💡 Research Summary

The paper tackles one of the most challenging problems in modern cyber‑forensics: detecting highly stealthy zero‑day malware that resides only in volatile memory. The authors begin by defining a “highest‑stealth” malware scenario, which combines all known anti‑forensic techniques—code obfuscation, packing, encryption, memory padding, and anticipated future improvements—into a single, hard‑to‑detect prototype. To confront this, they propose a multi‑layered analysis pipeline that fuses statistical, image‑processing, and code‑disassembly methods, and they accelerate the entire workflow using CUDA‑enabled GPUs.

The first analytical layer computes Shannon entropy for each memory page and visualizes the entropy map as a high‑resolution grid. While entropy alone is insufficient to spot sophisticated padding or partial compression, the grid representation allows rapid identification of anomalous regions that deviate from the typical uniform distribution of legitimate code and data.

The second layer borrows techniques from digital photogrammetry. Memory pages are treated as grayscale images; feature detectors such as SIFT or SURF extract keypoints, and Gray‑Level Co‑occurrence Matrices (GLCM) provide texture descriptors. These texture features capture structural irregularities—e.g., unexpected block arrangements or non‑standard data layouts—that often accompany hidden malicious payloads.

The third layer applies the Zipf‑Mandelbrot law to the frequency distribution of disassembled instruction tokens and embedded strings. Normal binaries exhibit a power‑law distribution with predictable scale and offset parameters, whereas encrypted or random payloads produce markedly different parameters. By fitting each page’s distribution to the Zipf‑Mandelbrot model and measuring the fitting error, the system generates an “anomaly score” that is independent of signature‑based detection.

All three scores are combined into a composite risk metric for every page. Pages exceeding a configurable threshold are passed to a dynamic disassembly stage. Using an LLVM‑based decoder, the system identifies function boundaries, builds control‑flow graphs (CFGs), and compares them against a library of known malicious patterns (e.g., API call sequences, loop manipulation). The final verdict is derived from a weighted aggregation of entropy, texture, Zipf‑Mandelbrot, and CFG similarity scores.

The computational heart of the proposal is a CUDA‑based accelerator. Memory is partitioned into 256 KB blocks, each mapped to a CUDA thread block. Separate kernels compute entropy, texture descriptors, and Zipf‑Mandelbrot fitting in parallel, exploiting shared memory to reduce global memory traffic. The disassembly step is also parallelized, allowing thousands of pages to be processed simultaneously. Benchmarks on an 8 GB Windows memory dump show an average 12× speed‑up over a single‑core CPU (3.2 GHz) and a 4× improvement over an 8‑core configuration, while detection accuracy improves by roughly 18 % compared with traditional entropy‑only tools.

The authors acknowledge several limitations. The experimental dataset contains a limited number of zero‑day samples, and the GPU’s memory capacity and power consumption may hinder deployment on resource‑constrained forensic workstations. Moreover, adversaries could adapt by deliberately crafting memory access patterns that obscure entropy, texture, or frequency signatures, potentially neutralizing the proposed pipeline. Future work is outlined to include multi‑GPU clustering, FPGA‑GPU hybrid acceleration, and integration with real‑time memory capture streams, aiming to transform the prototype into a production‑ready forensic platform.