Analysis of a Stochastic Model of Replication in Large Distributed Storage Systems: A Mean-Field Approach

Distributed storage systems such as Hadoop File System or Google File System (GFS) ensure data availability and durability using replication. This paper is focused on the analysis of the efficiency of replication mechanism that determines the location of the copies of a given file at some server. The variability of the loads of the nodes of the network is investigated for several policies. Three replication mechanisms are tested against simulations in the context of a real implementation of a such a system: Random, Least Loaded and Power of Choice. The simulations show that some of these policies may lead to quite unbalanced situations: if $\beta$ is the average number of copies per node it turns out that, at equilibrium, the load of the nodes may exhibit a high variability. It is shown in this paper that a simple variant of a power of choice type algorithm has a striking effect on the loads of the nodes: at equilibrium, the distribution of the load of a node has a bounded support, most of nodes have a load less than $2\beta$ which is an interesting property for the design of the storage space of these systems. Mathematical models are introduced and investigated to explain this interesting phenomenon. The analysis of these systems turns out to be quite complicated mainly because of the large dimensionality of the state spaces involved. Our study relies on probabilistic methods, mean-field analysis, to analyze the asymptotic behavior of an arbitrary node of the network when the total number of nodes gets large. An additional ingredient is the use of stochastic calculus with marked Poisson point processes to establish some of our results.

💡 Research Summary

The paper investigates how different data‑replication placement policies affect load balancing in large‑scale distributed storage systems such as Hadoop Distributed File System (HDFS) and Google File System (GFS). Three policies are examined: Random (uniformly select a storage node), Least Loaded (choose the node with the smallest current load), and Power of Choice (sample two nodes uniformly at random and store the replica on the less loaded of the two). The authors implement a realistic simulation using the PeerSim framework, modeling 200 storage nodes, 10 000 data blocks, a replication factor of three, and node failures occurring as a Poisson process with a mean time between failures of seven days. Over a simulated period of two years, they record the evolution of per‑node storage load under each policy.

Simulation results show that the Least Loaded policy keeps node load essentially constant at the optimal value (average load β = 150 blocks per node) but requires global knowledge of all node loads, leading to high communication overhead and potential network congestion when a lightly loaded node receives many repair transfers. The Random policy leads to a linear increase in load over a node’s lifetime, implying that older nodes accumulate many replicas and that their failure triggers costly recovery operations. The Power of Choice policy exhibits a much slower load growth; the distribution of node loads remains tightly concentrated and, remarkably, appears bounded by 2β even for moderate β values.

To explain these observations, the authors develop stochastic models for each policy using marked Poisson point processes. For the Random policy, they prove that as the number of nodes N → ∞, the load X_N of a typical node, normalized by β, converges in distribution to an exponential law: P(X_N/β ≥ x) → e^{−x}. For the Power of Choice policy, a more intricate mean‑field analysis yields a fixed‑point equation whose solution shows that the normalized load Y = X/β is confined to the interval