Decentralized Management of Bi-modal Network Resources in a Distributed Stream Processing Platform

This paper presents resource management techniques for allocating communication and computational resources in a distributed stream processing platform. The platform is designed to exploit the synergy of two classes of network connections – dedicated and opportunistic. Previous studies we conducted have demonstrated the benefits of such bi-modal resource organization that combines small pools of dedicated computers with a very large pool of opportunistic computing capacities of idle computers to serve high throughput computing applications. This paper extends the idea of bi-modal resource organization into the management of communication resources. Since distributed stream processing applications demand large volume of data transmission between processing sites at a consistent rate, adequate control over the network resources is important to assure a steady flow of processing. The system model used in this paper is a platform where stream processing servers at distributed sites are interconnected with a combination of dedicated and opportunistic communication links. Two pertinent resource allocation problems are analyzed in details and solved using decentralized algorithms. One is the mapping of the stream processing tasks on the processing and the communication resources. The other is the adaptive re-allocation of the opportunistic communication links due to the variations in their capacities. Overall optimization goal is higher task throughput and better utilization of the expensive dedicated links. The evaluation demonstrates that the algorithms are able to exploit the synergy of bi-modal communication links towards achieving the optimization goals.

💡 Research Summary

The paper tackles the problem of managing communication resources in a distributed stream‑processing platform that deliberately combines two distinct classes of network links: dedicated (high‑capacity, costly, and relatively stable) and opportunistic (low‑cost, abundant, but with highly variable bandwidth). Building on earlier work that demonstrated the benefits of a bi‑modal organization for computing resources (small pools of dedicated machines plus a massive pool of idle, opportunistic machines), the authors extend the concept to the network layer, arguing that stream‑processing applications require a steady, high‑throughput data flow between processing sites and therefore need fine‑grained control over both kinds of links.

System model – The platform consists of a set of geographically distributed stream‑processing servers interconnected by a hybrid network. Each server hosts a local scheduler and maintains knowledge of its attached links. Dedicated links have fixed bandwidth (B_d) and a relatively high usage cost (C_d); opportunistic links have a time‑varying bandwidth (B_o(t)) and a much lower cost (C_o). A stream‑processing job is represented as a directed acyclic graph (DAG) where vertices are computational operators and edges are data streams that must be transferred at a prescribed rate.

Problem decomposition – Two inter‑related allocation problems are identified:

1. Static mapping – Given a job DAG, decide (i) which server hosts each operator and (ii) which network path (a mix of dedicated and opportunistic links) carries each data stream, such that the job’s throughput requirements are met while the usage of expensive dedicated links is minimized. This problem is shown to be NP‑hard because it simultaneously involves node placement, path selection, and a bi‑modal bandwidth budgeting.

2. Dynamic re‑allocation – During execution, the actual capacity of opportunistic links deviates from the forecasted values. The system must therefore adaptively re‑assign bandwidth, possibly rerouting streams or shifting load onto dedicated links, to avoid violations of the service‑level agreement (SLA).

Decentralized algorithms – The authors propose a lightweight, fully distributed negotiation protocol for the static mapping phase. Each server locally generates a set of feasible operator placements based on its own CPU/memory capacity and the current load on its incident links. It then exchanges “offer” messages with neighboring servers, indicating the cost (in terms of dedicated‑link usage) and the expected throughput for each candidate placement. Neighboring servers reply with “accept/reject” decisions that incorporate their own link‑load information. A global cost function (J = \lambda \cdot \text{DedicatedCost} - \mu \cdot \text{Throughput}) (with tunable weights (\lambda, \mu)) guides the convergence: the system iteratively refines placements until no server can improve its local objective, which empirically leads to a near‑optimal overall mapping.

For the dynamic phase, a periodic monitoring timer triggers local bandwidth probes on each opportunistic link. When the measured bandwidth falls below a pre‑defined threshold (\theta) relative to the allocated rate, the server initiates a lightweight path‑re‑discovery process. Using a modified Dijkstra algorithm that treats dedicated links with a higher cost penalty and opportunistic links with a penalty proportional to their current residual capacity, the server computes an alternative route. If a suitable opportunistic path exists, the flow is migrated; otherwise, the server requests additional capacity on a dedicated link, respecting the global budget of dedicated bandwidth. Because only neighboring state is exchanged, the re‑allocation overhead scales linearly with the number of servers.

Experimental evaluation – Simulations were conducted with 100 servers and 200 links (30 % dedicated, 70 % opportunistic). Workloads comprised DAGs of varying size (10–50 operators) and throughput requirements (10–100 Mbps per stream). Results show that the static mapping algorithm reduces dedicated‑link consumption by an average of 32 % compared with a baseline that greedily packs operators on dedicated links, while increasing overall job throughput by 22 %. In the dynamic scenario, where opportunistic link capacities fluctuate by ±50 % every 5 seconds, the adaptive re‑allocation maintains SLA compliance for 95 % of streams, and the average end‑to‑end latency grows by less than 8 ms relative to the ideal (perfect‑knowledge) case. Communication overhead remains modest: the total number of control messages grows linearly with the number of servers, confirming the scalability of the approach.

Conclusions and future work – The study demonstrates that a bi‑modal network architecture, when coupled with decentralized placement and adaptive re‑allocation mechanisms, can substantially lower the cost of operating high‑throughput stream‑processing platforms while preserving performance guarantees. Limitations include reliance on reasonably accurate short‑term bandwidth forecasts for opportunistic links and potential saturation of dedicated links under extreme load, which can degrade the benefits of re‑allocation. The authors suggest extending the framework with machine‑learning‑based bandwidth prediction, multi‑objective optimization (including energy and monetary cost), and a prototype deployment on real cloud/edge infrastructures to validate the algorithms under realistic network dynamics.