Accelerating Large-scale Data Exploration through Data Diffusion
📝 Abstract
Data-intensive applications often require exploratory analysis of large datasets. If analysis is performed on distributed resources, data locality can be crucial to high throughput and performance. We propose a “data diffusion” approach that acquires compute and storage resources dynamically, replicates data in response to demand, and schedules computations close to data. As demand increases, more resources are acquired, thus allowing faster response to subsequent requests that refer to the same data; when demand drops, resources are released. This approach can provide the benefits of dedicated hardware without the associated high costs, depending on workload and resource characteristics. The approach is reminiscent of cooperative caching, web-caching, and peer-to-peer storage systems, but addresses different application demands. Other data-aware scheduling approaches assume dedicated resources, which can be expensive and/or inefficient if load varies significantly. To explore the feasibility of the data diffusion approach, we have extended the Falkon resource provisioning and task scheduling system to support data caching and data-aware scheduling. Performance results from both micro-benchmarks and a large scale astronomy application demonstrate that our approach improves performance relative to alternative approaches, as well as provides improved scalability as aggregated I/O bandwidth scales linearly with the number of data cache nodes.
💡 Analysis
Data-intensive applications often require exploratory analysis of large datasets. If analysis is performed on distributed resources, data locality can be crucial to high throughput and performance. We propose a “data diffusion” approach that acquires compute and storage resources dynamically, replicates data in response to demand, and schedules computations close to data. As demand increases, more resources are acquired, thus allowing faster response to subsequent requests that refer to the same data; when demand drops, resources are released. This approach can provide the benefits of dedicated hardware without the associated high costs, depending on workload and resource characteristics. The approach is reminiscent of cooperative caching, web-caching, and peer-to-peer storage systems, but addresses different application demands. Other data-aware scheduling approaches assume dedicated resources, which can be expensive and/or inefficient if load varies significantly. To explore the feasibility of the data diffusion approach, we have extended the Falkon resource provisioning and task scheduling system to support data caching and data-aware scheduling. Performance results from both micro-benchmarks and a large scale astronomy application demonstrate that our approach improves performance relative to alternative approaches, as well as provides improved scalability as aggregated I/O bandwidth scales linearly with the number of data cache nodes.
📄 Content
The ability to analyze large quantities of data has become increasingly important in many fields. To achieve rapid turnaround, data may be distributed over hundreds of computers. In such circumstances, data locality has been shown to be crucial to the successful and efficient use of large distributed systems for data-intensive applications [7,34].
One approach to achieving data locality-adopted, for example, by Google [3,11]-is to build large compute-storage farms dedicated to storing data and responding to user requests for processing. However, such approaches can be expensive (in terms of idle resources) if load varies significantly over the two dimensions of time and/or the data of interest.
This paper proposes an alternative data diffusion approach, in which resources required for data analysis are acquired dynamically, in response to demand. Resources may be acquired either “locally” or “remotely”; their location only matters in terms of associated cost tradeoffs. Both data and applications are copied (they “diffuse”) to newly acquired resources for processing. Acquired resources (computers and storage) and the data that they hold can be “cached” for some time, thus allowing more rapid responses to subsequent requests. If demand drops, resources can be released, allowing their use for other purposes. Thus, data diffuses over an increasing number of CPUs as demand increases, and then contracting as load reduces.
Data diffusion thus involves a combination of dynamic resource provisioning, data caching, and data-aware scheduling. The approach is reminiscent of cooperative caching [18], cooperative web-caching [19], and peer-to-peer storage systems [17]. (Other data-aware scheduling approaches tend to assume static resources [1,2].) However, in our approach we need to acquire dynamically not only storage resources but also computing resources. In addition, datasets may be terabytes in size and data access is for analysis (not retrieval). Further complicating the situation is our limited knowledge of workloads, which may involve many different applications.
In our exploration of these issues, we build upon previous work on Falkon, a Fast and Light-weight tasK executiON framework [4,12], which provides for dynamic acquisition and release of resources (“workers”) and the dispatch of analysis tasks to those workers. We describe Falkon data caching extensions that enable (in their current instantiation) the management of tens of millions of files spanning hundreds of multiple storage resources.
In principle, data diffusion can provide the benefit of dedicated hardware without the associated high costs. It can also overcome inefficiencies that may arise when executing data-intensive applications in distributed (“grid”) environments, due to the high costs of data movement [34]: if workloads have sufficient internal locality of reference [22], then it is feasible to acquire and use even remote resources despite high initial data movement costs.
The performance achieved with data diffusion depends crucially on the precise characteristics of application workloads and the underlying infrastructure. As a first step towards quantifying these dependences, we have conducted experiments with both microbenchmarks and a large scale astronomy application. The experiments presented here do not investigate the effects of dynamic resource provisioning, which we will address in future work. They show that our approach improves performance relative to alternative approaches, and provides improved scalability as aggregated I/O bandwidth scales linearly with the number of data cache nodes.
The results presented here build on our past work on resource provisioning [12] and task dispatching [4], and implement ideas outlined in a previous short paper [26].
Data management becomes more useful if coupled with compute resource management. Ranganathan et al. used simulation studies [10] to show that proactive data replication can improve application performance. The Stork [28] scheduler seeks to improve performance and reliability when batch scheduling by explicitly scheduling data placement operations. However, while Stork can be used with other system components to co-schedule CPU and storage resources, there is no attempt to retain nodes between tasks as in our work.
The GFarm team implemented a data-aware scheduler in Gfarm using an LSF scheduler plugin [1,23]. Their performance results are for a small system (6 nodes, 300 jobs, 900 MB input files, 2640 second workload without data-aware scheduling, 1650 seconds with data-aware scheduling, 0.1-0.2 jobs/sec, 90MB/s to 180MB/s data rates); it is not clear that it scales to larger systems. In contrast, we have tested our proposed data diffusion with 64 nodes, 100K jobs, input data ranging from 1B to 1GB, workflows exceeding 1000 jobs/sec, and data rates exceeding 8750 MB/s. BigTable [21], Google File System (GFS) [3], and MapReduce [11] (or the open source implementation in Hado
This content is AI-processed based on ArXiv data.