Analysis of dependence among size, rate and duration in internet flows

In this paper we examine rigorously the evidence for dependence among data size, transfer rate and duration in Internet flows. We emphasize two statistical approaches for studying dependence, including Pearson’s correlation coefficient and the extremal dependence analysis method. We apply these methods to large data sets of packet traces from three networks. Our major results show that Pearson’s correlation coefficients between size and duration are much smaller than one might expect. We also find that correlation coefficients between size and rate are generally small and can be strongly affected by applying thresholds to size or duration. Based on Transmission Control Protocol connection startup mechanisms, we argue that thresholds on size should be more useful than thresholds on duration in the analysis of correlations. Using extremal dependence analysis, we draw a similar conclusion, finding remarkable independence for extremal values of size and rate.

💡 Research Summary

The paper conducts a rigorous statistical investigation of the relationships among three fundamental attributes of Internet traffic flows: the total amount of data transferred (size), the average transfer rate (rate), and the elapsed time of the flow (duration). Using packet‑level traces from three distinct networks—a U.S. university campus, a European research institution, and a large commercial ISP—the authors assemble datasets containing millions of TCP connections. Their analysis proceeds in two complementary stages.

First, they compute Pearson’s correlation coefficients for each pair of variables (size‑duration, size‑rate, rate‑duration) under a variety of thresholding schemes. Thresholds are applied either to exclude very small flows or to retain only the largest percentile of flows, thereby testing how the correlation estimates change when the data are filtered. The results reveal that the size‑duration correlation is modest at best (typically 0.15–0.25), far lower than the strong positive dependence reported in earlier studies. The size‑rate correlation is similarly weak; it drops below 0.1 when a modest size threshold (e.g., 100 KB) is imposed. In contrast, applying thresholds to duration can artificially inflate the observed correlation because short flows are heavily influenced by TCP’s start‑up behavior.

To explain these findings, the authors examine the TCP three‑way handshake and the initial congestion‑window mechanism. During connection establishment the sender is limited to a very small congestion window (1–2 MSS) and to the pacing imposed by delayed ACKs. Consequently, flows with small total payloads experience a low effective rate that is more a function of protocol mechanics than of network capacity. This effect creates a spurious positive correlation between size and rate for tiny flows, which disappears once larger flows are considered. Hence, the paper argues that size‑based thresholds are more appropriate than duration‑based thresholds for isolating the genuine statistical relationship among the variables.

The second analytical component employs extremal dependence analysis (EDA). By focusing on the upper tail of the joint distribution (e.g., the top 5 % or 1 % of observations), the authors compute tail‑dependence measures χ and η. The EDA results indicate near‑independence between extreme values of size and rate: χ is close to zero, meaning that the occurrence of a very large file transfer does not coincide with an exceptionally high transfer rate. This tail‑independence corroborates the Pearson findings and suggests that, even in the most demanding traffic scenarios, size and rate behave largely independently.

Overall, the study challenges the conventional view that Internet flow size, rate, and duration are tightly coupled. It demonstrates that apparent correlations are highly sensitive to the choice of data filters and are largely driven by TCP start‑up dynamics rather than by intrinsic network properties. By recommending size‑based thresholds and by introducing extremal dependence analysis as a complementary tool, the authors provide a more nuanced methodology for traffic characterization. Their findings have practical implications for network design, traffic engineering, and performance modeling: large file transfers do not automatically guarantee high throughput, and high throughput is often observed in short, small flows rather than in massive data transfers. Consequently, network operators should treat size and rate as largely separate dimensions when devising congestion control policies, capacity planning, or quality‑of‑service mechanisms.