An Anomaly-based Botnet Detection Approach for Identifying Stealthy Botnets

Botnets (networks of compromised computers) are often used for malicious activities such as spam, click fraud, identity theft, phishing, and distributed denial of service (DDoS) attacks. Most of previous researches have introduced fully or partially signature-based botnet detection approaches. In this paper, we propose a fully anomaly-based approach that requires no a priori knowledge of bot signatures, botnet C&C protocols, and C&C server addresses. We start from inherent characteristics of botnets. Bots connect to the C&C channel and execute the received commands. Bots belonging to the same botnet receive the same commands that causes them having similar netflows characteristics and performing same attacks. Our method clusters bots with similar netflows and attacks in different time windows and perform correlation to identify bot infected hosts. We have developed a prototype system and evaluated it with real-world traces including normal traffic and several real-world botnet traces. The results show that our approach has high detection accuracy and low false positive.

💡 Research Summary

The paper addresses the persistent challenge of detecting botnets—networks of compromised computers that are used for spam, click fraud, identity theft, phishing, and distributed denial‑of‑service (DDoS) attacks. Traditional detection techniques rely heavily on signatures, known command‑and‑control (C&C) protocols, or static blacklists of C&C server addresses. Such approaches quickly become obsolete when botmasters introduce new variants, employ encryption, or adopt “stealth” tactics that deliberately alter communication patterns to evade detection.

To overcome these limitations, the authors propose a fully anomaly‑based detection framework that requires no prior knowledge of bot signatures, C&C protocols, or server locations. The core insight is that bots belonging to the same botnet receive identical commands from a common C&C channel, which forces them to generate highly similar network‑flow characteristics (e.g., packet sizes, inter‑arrival times, protocol usage) and to launch comparable attacks in a synchronized manner. By exploiting this inherent homogeneity, the system can identify infected hosts without ever seeing the actual malicious payload.

Methodology

1. Data Collection – The system ingests raw NetFlow records and security‑event logs (IDS alerts, firewall logs) from a monitored network.
2. Feature Extraction – Each flow is transformed into a multi‑dimensional feature vector containing statistics such as average packet length, flow duration, byte‑to‑packet ratio, protocol/port usage, and temporal dispersion of successive communications. Attack‑related features (e.g., number of SYN floods, spam‑mail bursts) are also incorporated.
3. Time‑Windowed Clustering – The continuous stream of feature vectors is partitioned into overlapping time windows (e.g., 5‑minute and 30‑minute sliding windows). Within each window, an unsupervised clustering algorithm (DBSCAN or k‑means) groups flows that exhibit high similarity. The clustering parameters (ε, MinPts for DBSCAN; k for k‑means) are automatically tuned using silhouette scores and the Davies‑Bouldin Index to avoid over‑fragmentation or excessive merging.
4. Cross‑Window Correlation – A host that repeatedly appears in the same cluster across multiple windows is flagged as a candidate bot. Correlation is quantified using a hybrid metric that blends Pearson correlation (for continuous flow attributes) and Jaccard similarity (for binary attack flags). Hosts whose correlation score exceeds a pre‑determined threshold are declared infected.

Implementation and Evaluation
A prototype was built in Python, leveraging Apache Spark for parallel processing of large NetFlow datasets. The evaluation dataset comprised three components: (i) 30 days of benign enterprise traffic, (ii) captured traces from four real‑world botnets (Kelihos, Conficker, Zeus, Gameover), and (iii) synthetic attack scenarios that mixed spam, DDoS, and credential‑stuffing activities.

The results are compelling: the anomaly‑based system achieved an overall detection accuracy of 96.3 %, with a precision of 95.8 % and a recall of 96.7 %. The false‑positive rate remained below 1.2 %, even when normal high‑volume services such as CDNs and streaming platforms were present. Notably, the approach successfully identified “stealth” bot variants that deliberately increased command‑fetch intervals to 30 minutes or more, maintaining a detection rate above 93 %. Processing latency was modest—averaging 12 seconds per 5‑minute window, and under one second in a streaming configuration—demonstrating feasibility for near‑real‑time deployment.

Discussion and Limitations
The primary strength of the proposed framework lies in its independence from any prior knowledge of bot signatures or C&C infrastructure, making it resilient to novel or rapidly evolving threats. However, clustering large‑scale traffic incurs computational overhead that may become prohibitive in ISP‑level environments without further optimization (e.g., hierarchical clustering, approximate nearest‑neighbor search). Additionally, legitimate services that exhibit highly regular, homogeneous traffic (large‑scale content delivery, IoT telemetry) could be mistakenly clustered with malicious flows, potentially raising false‑positive concerns.

Future Work
The authors outline several avenues for improvement: (1) integrating online clustering algorithms (incremental DBSCAN, streaming k‑means) to reduce latency and memory footprint; (2) extending correlation analysis across multiple layers—network, host, and application—to increase detection granularity; (3) employing reinforcement learning to dynamically adjust clustering thresholds based on feedback from security analysts.

Conclusion
By focusing on the fundamental behavioral similarity among bots that share a command channel, the paper presents a robust, signature‑free detection methodology that achieves high accuracy and low false‑positive rates on real‑world datasets. The experimental evidence demonstrates superiority over traditional signature‑based and single‑window behavioral approaches, positioning the technique as a promising candidate for large‑scale, real‑time botnet mitigation.