This paper presents resource management techniques for allocating communication and computational resources in a distributed stream processing platform. The platform is designed to exploit the synergy of two classes of network connections -- dedicated and opportunistic. Previous studies we conducted have demonstrated the benefits of such bi-modal resource organization that combines small pools of dedicated computers with a very large pool of opportunistic computing capacities of idle computers to serve high throughput computing applications. This paper extends the idea of bi-modal resource organization into the management of communication resources. Since distributed stream processing applications demand large volume of data transmission between processing sites at a consistent rate, adequate control over the network resources is important to assure a steady flow of processing. The system model used in this paper is a platform where stream processing servers at distributed sites are interconnected with a combination of dedicated and opportunistic communication links. Two pertinent resource allocation problems are analyzed in details and solved using decentralized algorithms. One is the mapping of the stream processing tasks on the processing and the communication resources. The other is the adaptive re-allocation of the opportunistic communication links due to the variations in their capacities. Overall optimization goal is higher task throughput and better utilization of the expensive dedicated links. The evaluation demonstrates that the algorithms are able to exploit the synergy of bi-modal communication links towards achieving the optimization goals.
Deep Dive into Decentralized Management of Bi-modal Network Resources in a Distributed Stream Processing Platform.
This paper presents resource management techniques for allocating communication and computational resources in a distributed stream processing platform. The platform is designed to exploit the synergy of two classes of network connections – dedicated and opportunistic. Previous studies we conducted have demonstrated the benefits of such bi-modal resource organization that combines small pools of dedicated computers with a very large pool of opportunistic computing capacities of idle computers to serve high throughput computing applications. This paper extends the idea of bi-modal resource organization into the management of communication resources. Since distributed stream processing applications demand large volume of data transmission between processing sites at a consistent rate, adequate control over the network resources is important to assure a steady flow of processing. The system model used in this paper is a platform where stream processing servers at distributed sites are in
Many applications on the Internet are creating, manipulating, and consuming data at an astonishing rate. Data stream processing is one such class of applications where data is streamed through a network of servers that operate on the data as they pass through them [1,2,3,4,5,6,7]. Depending on the application, data streams can have complex topologies with multiple sources or multiple sinks. Examples of data stream processing tasks are found in many areas including distributed databases, sensor networks, and multimedia computing. Some examples include: (i) multimedia streams of real-time events that are transcoded into different formats [8], (ii) insertion of information tickers into multimedia streams [9], (iii) real-time analysis of network monitoring data streams for malicious activity detection [10], and (iv) function computation over data feeds obtained from sensor networks [4].
One of the salient characteristics of this class of applications is the simultaneous demand for high-throughput computing and communication resources [11]. Huge volume of data generated at high rates need to be processed within real-time constraints. Moreover, various operations on these data streams are provided by different servers at distributed geographic locations [12]. All these factors demand a scalable and adaptive architecture for distributed stream processing platform, where fine-grained control over processing and network resources is possible.
Earlier works on stream processing engines [13,14] resorted to centralized single-server or server-cluster based solutions where tighter control over available resources is possible. With the possibility of different processing services or operations being provided by different providers, need for distributed stream processing platform arose. Several architectures have been proposed to support such distributed processing of streams [11,15,12,16]. Due to the stringent rate-requirement for processing and transmission of data, most researchers have assumed a central resource controller that can gather the availability status of all re-sources and map the requested tasks on them. However, with the advent of a diverse range of stream processing services, it is important to allow autonomous providers of services to collaborate and share their resources. Thus it is important to develop decentralized resource allocation schemes, where control is available over local resources only.
While it is feasible to have dedicated server resources and precisely allocate them for processing tasks, dedicated networks over wide-area installations remain costly. Although it is possible to propagate the data streams through the distributed servers using the Internet, the lack of adequate control over end-to-end bandwidth on the Internet and the stringent rate requirements of the stream processing applications demand some dedicated network resources. In fact, recent advances in optical network technologies such as user-controlled light path [17,18] open the possibility of on-demand provisioning of end-to-end optical links with total control of the available bandwidth is exposed to the user application.
In this paper, we explore a novel approach where a combination of dedicated and opportunistic communication links is used to interconnect the servers. The main focus of this paper is to explore how such a hybrid (denoted as bi-modal in this paper) network can be best used for data stream processing tasks. The hypothesis that drives this work is that the combination has a synergistic effect that allows better utilization of the dedicated resources, and yields higher return on investment. We devised distributed algorithms for allocation of these hybrid resources to demonstrate the viability of this synergy hypothesis.
Multiple global objectives such as higher task throughput, lower violation of SLA and higher utilization of dedicated resources make the resource management a complex task, especially when allocation decisions are to be taken solely based on the local information available on the server nodes. We divided the overall resource management process into two steps -initially individual tasks are assigned node and link resources through a distributed mapping algorithm. Based on actual resource availability, link resources are then periodically re-allocated locally among competing tasks towards the global optimization objectives.
This paper extends some of our previous works [19,20] on bi-modal compute platforms where static small pool of dedicated compute-servers was combined with a large number of opportunistically harvested cheap processing elements to increase work throughput and utilization of dedicated resources. Using data stream processing tasks as a concrete example, this paper demonstrates the benefit of using bi-modal network infrastructures for communication-intensive applications. In particular, this paper makes the following contributions to this important resource management prob
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
