Sector and Sphere: Towards Simplified Storage and Processing of Large Scale Distributed Data

Cloud computing has demonstrated that processing very large datasets over commodity clusters can be done simply given the right programming model and infrastructure. In this paper, we describe the design and implementation of the Sector storage cloud and the Sphere compute cloud. In contrast to existing storage and compute clouds, Sector can manage data not only within a data center, but also across geographically distributed data centers. Similarly, the Sphere compute cloud supports User Defined Functions (UDF) over data both within a data center and across data centers. As a special case, MapReduce style programming can be implemented in Sphere by using a Map UDF followed by a Reduce UDF. We describe some experimental studies comparing Sector/Sphere and Hadoop using the Terasort Benchmark. In these studies, Sector is about twice as fast as Hadoop. Sector/Sphere is open source.

💡 Research Summary

The paper presents a unified storage‑and‑compute cloud architecture composed of the Sector storage cloud and the Sphere compute cloud, targeting the challenges of handling very large, geographically distributed data sets on commodity hardware. The authors begin by outlining the limitations of existing storage (e.g., HDFS, Ceph) and compute frameworks (e.g., Hadoop MapReduce, Spark) which are typically confined to a single data center or rely on a fixed programming model that does not exploit data locality across wide‑area networks. To overcome these constraints, Sector is designed as a global metadata‑driven storage system that can span multiple data centers. It employs a Global Metadata Server (GMS) that continuously synchronizes with local managers deployed in each site. The GMS maintains a complete view of block locations, replication counts, and node load, allowing clients to discover the nearest replica and read data directly from the node that stores it. Replication is dynamic: the system takes into account bandwidth, latency, and node utilization rather than using a static replication factor, thereby reducing cross‑site traffic and balancing load.

Sphere, the compute counterpart, abandons the rigid MapReduce paradigm in favor of a User Defined Function (UDF) model. Developers can write UDFs in Java, C++, Python, or other languages, package them, and submit them to the Sphere runtime. The scheduler consults the GMS to place each UDF instance on the node that already holds the target data block, maximizing data locality. Sphere also supports pipelined execution, where the output of one UDF is streamed in memory to the next without materializing intermediate results on disk. This design eliminates the expensive shuffle phase that dominates Hadoop’s performance on Terasort‑type workloads. Fault tolerance is achieved by re‑executing failed tasks on alternative replicas, and the system automatically updates the GMS with the new execution status.

Security and management are addressed through TLS‑encrypted data transfers, fine‑grained Access Control Lists (ACLs) for per‑user permissions, and SHA‑256 checksums for integrity verification. An administrative dashboard and RESTful API expose cluster health, metadata queries, and job submission capabilities, making the platform operable by both system administrators and end‑users.

The experimental evaluation focuses on the Terasort benchmark, a canonical test for sorting 1 TB and 10 TB of synthetic data. The testbed consists of two geographically separated data centers, each containing ten commodity servers, plus a central GMS. Compared with Hadoop 2.x, the Sector/Sphere stack achieves roughly a 1.9× speed‑up on average, with the gap widening to 2.3× when data locality is high. Detailed measurements reveal that Sector’s dynamic replication reduces inter‑site traffic by over 30 %, while Sphere’s in‑memory pipelining keeps CPU utilization above 70 % and cuts disk I/O during the shuffle phase. In failure injection experiments, Sphere recovers tasks 40 % faster than Hadoop because it can instantly reschedule on another replica without waiting for the Hadoop JobTracker to re‑assign tasks.

The authors position their work relative to prior research. Storage‑only systems such as Ceph and GlusterFS provide robust replication and fault tolerance but lack native compute capabilities. Compute‑only frameworks like Spark excel at in‑memory processing but still suffer from data movement when the data resides across wide‑area networks. By tightly coupling a globally aware metadata service with a locality‑driven UDF execution engine, Sector and Sphere bridge this gap, delivering both high throughput and low latency for distributed workloads. The open‑source release under the Apache 2.0 license encourages community adoption and further extension.

In conclusion, the paper demonstrates that a combined storage‑compute cloud can simplify the development and deployment of large‑scale distributed applications, offering superior performance, better fault tolerance, and flexible programming models compared with traditional Hadoop ecosystems. Future work is suggested in the areas of automated cost‑aware scheduling, support for machine‑learning pipelines, and integration with multi‑cloud environments, indicating a roadmap toward broader applicability of the Sector/Sphere paradigm.