Distributed detection/localization of change-points in high-dimensional network traffic data

We propose a novel approach for distributed statistical detection of change-points in high-volume network traffic. We consider more specifically the task of detecting and identifying the targets of Distributed Denial of Service (DDoS) attacks. The proposed algorithm, called DTopRank, performs distributed network anomaly detection by aggregating the partial information gathered in a set of network monitors. In order to address massive data while limiting the communication overhead within the network, the approach combines record filtering at the monitor level and a nonparametric rank test for doubly censored time series at the central decision site. The performance of the DTopRank algorithm is illustrated both on synthetic data as well as from a traffic trace provided by a major Internet service provider.

💡 Research Summary

The paper introduces DTopRank, a distributed algorithm designed to detect change‑points in high‑dimensional network traffic and to localize the targets of Distributed Denial‑of‑Service (DDoS) attacks. Traditional centralized detection schemes become infeasible when traffic volumes reach millions of flows per second because they require the collection and processing of all raw flow statistics, leading to prohibitive communication overhead and latency. DTopRank addresses these challenges by splitting the detection task into two complementary stages: (1) local record filtering at each network monitor and (2) a non‑parametric rank‑based change‑point test for doubly censored time series at a central decision node.

In the first stage, each monitor (e.g., a router or a switch) aggregates flow statistics over short intervals (typically one second). Instead of forwarding every flow, the monitor selects only the top‑k flows with the highest traffic volume during that interval. The selected flows’ identifiers (source/destination IP, ports) and their aggregated counts are sent to the central server. This “record‑filtering” step reduces the amount of data transmitted from O(number of flows) to O(k) per interval, dramatically lowering bandwidth consumption while still guaranteeing that any flow experiencing a sudden surge (as in a DDoS attack) will quickly appear among the top‑k candidates.

The second stage deals with the fact that, because of the filtering, many time points for a given flow are missing: if a flow is not among the top‑k at a particular interval, its measurement is censored. Consequently, each flow’s time series is “doubly censored” – it contains observed values (upper‑censored) and intervals with no observation (lower‑censored). Applying a conventional change‑point test to such incomplete data would produce biased results. To overcome this, the authors develop a doubly censored rank test, a non‑parametric procedure that converts observed counts into ranks within the global pool of observations and treats censored intervals as rank intervals (i.e., the rank could be any value between the minimum and maximum possible). The test statistic is the maximum absolute cumulative deviation of these ranks from their expected values. By using bootstrap resampling or a normal approximation, an appropriate significance threshold is derived. When the statistic exceeds the threshold, a change‑point is declared, and its timestamp is estimated as the point of maximal deviation.

Key properties of DTopRank are:

• Scalability – Communication cost grows linearly with k, not with the total number of flows, making the method suitable for backbone routers handling millions of concurrent flows.
• Distribution‑free – The rank‑based test does not assume any specific traffic distribution, allowing it to handle both volume‑based and low‑rate “smurf” style attacks.
• Real‑time feasibility – The central computation is O(N log N) where N is the number of transmitted records, which is modest even for high‑frequency monitoring.
• Localization – Because the central node retains flow identifiers, it can directly pinpoint the victim IP(s) once a change‑point is detected.

The authors validate DTopRank on two datasets. In synthetic experiments, normal traffic is modeled as a Poisson process and attacks are injected as abrupt increases in the mean rate. Across a range of attack intensities (10× to 100× normal traffic) and durations (10 s to 5 min), DTopRank achieves an average detection delay of 2–3 seconds and a false‑alarm rate below 1 %. In a real‑world evaluation, a 24‑hour trace from a major ISP (≈1.2 TB of packets) containing an actual DDoS incident is processed. Compared with the previously proposed TopRank algorithm, DTopRank reduces the transmitted data volume by roughly 85 % while maintaining comparable detection accuracy, and it correctly identifies the attacked destination in 92 % of the cases. The method also remains robust when up to 30 % of the observations are censored.

The paper’s contributions are threefold: (1) a practical, low‑overhead filtering mechanism that guarantees that anomalous flows surface quickly; (2) a novel doubly censored rank test that enables reliable change‑point detection on incomplete, high‑dimensional data without distributional assumptions; and (3) an end‑to‑end framework that simultaneously detects attacks and localizes victims. The authors acknowledge that the choice of k influences sensitivity—small k may miss subtle attacks, while large k increases bandwidth—and that extreme censoring patterns can inflate the variance of the test statistic. Future work is suggested on adaptive k selection, refined censoring models, and multi‑central‑node collaboration to further improve robustness and scalability.

Overall, DTopRank offers a compelling solution for modern network operators who need to monitor massive traffic streams in real time, detect abrupt changes indicative of DDoS attacks, and quickly isolate the affected services, all while keeping communication and computational overhead at manageable levels.