Fast Botnet Detection From Streaming Logs Using Online Lanczos Method

Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20% to 25% of PCA.

💡 Research Summary

The paper addresses the problem of detecting botnets in real‑time from high‑volume web‑server log streams. Traditional botnet detection methods either focus on single bots, rely on specialized data sources (e.g., CAPTCHA results, DNS traffic), or apply full Principal Component Analysis (PCA) whose cubic time complexity makes it impractical for sliding‑window streaming scenarios. The authors propose a novel framework that (1) converts each sliding time window of logs into a host‑request matrix X (rows = request types, columns = hosts), (2) maintains an online update of the host‑host correlation matrix C = (1/(m‑1)) X̃ᵀ X̃ where X̃ is the column‑wise centered and normalized version of X, and (3) extracts the dominant eigenvalue (principal weight) using an adapted Lanczos iteration instead of full eigen‑decomposition.

Key technical contributions are:

• Generalized online correlation matrix update – When the sliding window moves, rows and columns may be added, removed, or have changed values. The authors derive formulas that update the column means, standard deviations, and consequently the correlation matrix in O(n·Δ) time (Δ = number of rows added/removed), avoiding the O(m·n) cost of recomputing from scratch. This works even when new hosts appear or disappear, by inserting or deleting zero‑filled columns.

• Lanczos‑based fast PCA – Lanczos iteration builds a small tridiagonal matrix T_k whose eigenvalues converge to those of C. The paper introduces two innovations: (i) an early‑termination condition based on a theoretical error bound and the monotonic non‑decrease of the largest eigenvalue for symmetric matrices, and (ii) a dynamic choice of the iteration count k, stopping as soon as the estimated principal weight satisfies the error tolerance. This yields a sub‑cubic overall complexity while still delivering the largest eigenvalue with sufficient accuracy for detection.

The detection rule is simple: if the principal weight exceeds a pre‑set threshold (e.g., 0.75) the system raises a botnet alert. Because the method monitors the correlation among hosts rather than absolute request rates, it can detect coordinated low‑rate botnets that would be invisible to peak‑finding or outlier‑based single‑bot detectors.

Experimental evaluation uses Apache access logs from a real e‑commerce site, generating roughly 100 000 entries per 30 minutes. The authors test three sliding‑window lengths (5 min, 10 min, 30 min) and compare the runtime of the classic PCA (full eigen‑decomposition) against the Lanczos‑based approach. Across all settings, Lanczos requires only 20 %–25 % of the CPU time of PCA, effectively a 4–5× speed‑up, while detection accuracy (precision, recall) remains comparable or slightly better, especially for the smaller windows where the method retains sensitivity.

Strengths:

• Substantial reduction in computational cost makes real‑time deployment feasible.
• The approach relies solely on standard log fields, preserving privacy and avoiding packet‑capture overhead.
• Sliding‑window operation provides finer temporal granularity than fixed‑window schemes, enabling earlier alerts.

Limitations:

• In extremely sparse host‑request matrices, Lanczos convergence may slow, suggesting a need for sparse‑matrix‑specific variants.
• The current implementation assumes a relatively stable host dimension; frequent addition/removal of hosts incurs matrix resizing overhead.
• The detection threshold is manually tuned; integrating an automatic anomaly‑score learning component would improve usability.
• Only one feature (host‑request count) is used; incorporating additional log attributes (user‑agent, response code, session duration) could boost robustness.

Future directions proposed by the authors include: developing Lanczos adaptations for highly sparse data, automating threshold selection via statistical learning, and extending the framework to multi‑feature correlation analysis. Overall, the paper demonstrates that an online Lanczos method combined with efficient correlation‑matrix updates offers a practical, scalable solution for real‑time botnet detection from streaming logs, outperforming traditional PCA in both speed and applicability.