In-Network Redundancy Generation for Opportunistic Speedup of Backup

Erasure coding is a storage-efficient alternative to replication for achieving reliable data backup in distributed storage systems. During the storage process, traditional erasure codes require a unique source node to create and upload all the redundant data to the different storage nodes. However, such a source node may have limited communication and computation capabilities, which constrain the storage process throughput. Moreover, the source node and the different storage nodes might not be able to send and receive data simultaneously – e.g., nodes might be busy in a datacenter setting, or simply be offline in a peer-to-peer setting – which can further threaten the efficacy of the overall storage process. In this paper we propose an “in-network” redundancy generation process which distributes the data insertion load among the source and storage nodes by allowing the storage nodes to generate new redundant data by exchanging partial information among themselves, improving the throughput of the storage process. The process is carried out asynchronously, utilizing spare bandwidth and computing resources from the storage nodes. The proposed approach leverages on the local repairability property of newly proposed erasure codes tailor made for the needs of distributed storage systems. We analytically show that the performance of this technique relies on an efficient usage of the spare node resources, and we derive a set of scheduling algorithms to maximize the same. We experimentally show, using availability traces from real peer-to-peer applications as well as Google data center availability and workload traces, that our algorithms can, depending on the environment characteristics, increase the throughput of the storage process significantly (up to 90% in data centers, and 60% in peer-to-peer settings) with respect to the classical naive data insertion approach.

💡 Research Summary

The paper addresses a fundamental bottleneck in erasure‑coded backup systems: the source node must generate all redundant fragments and upload them to storage nodes, which limits throughput especially when the source or storage nodes have limited bandwidth, CPU, or intermittent availability. To overcome this, the authors propose “in‑network redundancy generation,” a method that lets storage nodes create new redundant fragments by exchanging partial information among themselves, thereby distributing the insertion load.

The key enabling technology is a class of locally repairable erasure codes, specifically Self‑Repairing Codes (SRC) and their homomorphic variant (HSRC). HSRC possesses a homomorphic property: for any two field elements a and b, the code polynomial satisfies p(a + b) = p(a) + p(b). Consequently, a fragment evaluated at α_k can be reconstructed by XOR‑ing the fragments evaluated at α_i and α_j whenever α_k = α_i + α_j. This yields a repair degree d = 2, meaning any missing fragment can be regenerated by contacting only two live nodes. Because the operation is a simple XOR, it can be performed locally without involving the original source.

The authors formalize the problem as a scheduling task: given time‑varying node availability, bandwidth, and compute capacity, decide which pairs of nodes should exchange fragments in each time slot so that (1) the total backup completion time is minimized and (2) the extra network usage stays within the spare capacity. They prove that finding the optimal schedule is NP‑hard, motivating the design of heuristic algorithms. Four heuristics are explored:

1. Random Pairing – randomly matches available nodes; simple but inefficient.
2. Greedy Max‑Util – always selects the pair that can generate the largest number of still‑missing fragments, maximizing immediate resource utilization.
3. Availability‑Aware – predicts future node up‑times (e.g., via a Markov model) and selects pairs that are expected to provide the most transmission opportunities over the horizon.
4. Hybrid Scheduler – dynamically switches among the above strategies based on current system state, using Greedy during tight windows and Availability‑Aware when the system is stable.

The evaluation uses two real‑world trace sets. The first consists of Google data‑center logs (node availability, CPU load, and network usage) spanning a month; the second contains availability traces from friend‑to‑friend and generic peer‑to‑peer applications. Simulations are performed with a (14, 8) HSRC configuration (75 % storage efficiency). Results show that in the data‑center scenario the Hybrid scheduler achieves an average throughput increase of 65 %–90 % compared with the naive source‑centric approach, while consuming roughly 45 % of the otherwise idle network bandwidth. In the P2P scenario, the Availability‑Aware and Hybrid methods yield 40 %–60 % throughput gains with about 38 % extra bandwidth usage. The source node’s CPU load drops below 30 % of its baseline, and overall network congestion does not increase noticeably.

The paper also discusses limitations. The effectiveness of the Availability‑Aware heuristic depends on accurate predictions of node up‑times; poor forecasts can degrade performance. HSRC’s parameter constraints (n ≤ 2^m − 1, k < n) limit its applicability to arbitrary system sizes, requiring careful parameter selection. Although the approach consumes additional bandwidth, it leverages spare capacity that would otherwise remain unused, so the cost impact is modest. Extending the technique to codes with higher repair degrees (d > 2) or to more complex locally repairable codes (e.g., Azure LRC) would require further algorithmic development.

In conclusion, the work demonstrates that by exploiting locally repairable erasure codes and appropriate scheduling, in‑network redundancy generation can dramatically accelerate backup insertion without sacrificing reliability. The authors suggest future directions such as integrating multi‑degree locally repairable codes, employing machine‑learning‑based availability forecasts, and building a prototype system to validate the approach in a live production environment.