Toward a Principled Framework for Benchmarking Consistency

Large-scale key-value storage systems sacrifice consistency in the interest of dependability (i.e., partition tolerance and availability), as well as performance (i.e., latency). Such systems provide eventual consistency,which—to this point—has been difficult to quantify in real systems. Given the many implementations and deployments of eventually-consistent systems (e.g., NoSQL systems), attempts have been made to measure this consistency empirically, but they suffer from important drawbacks. For example, state-of-the art consistency benchmarks exercise the system only in restricted ways and disrupt the workload, which limits their accuracy. In this paper, we take the position that a consistency benchmark should paint a comprehensive picture of the relationship between the storage system under consideration, the workload, the pattern of failures, and the consistency observed by clients. To illustrate our point, we first survey prior efforts to quantify eventual consistency. We then present a benchmarking technique that overcomes the shortcomings of existing techniques to measure the consistency observed by clients as they execute the workload under consideration. This method is versatile and minimally disruptive to the system under test. As a proof of concept, we demonstrate this tool on Cassandra.

💡 Research Summary

The paper addresses a fundamental gap in the evaluation of eventually‑consistent key‑value stores: existing benchmarks either constrain the workload or disturb the system, leading to inaccurate measurements of the consistency that clients actually experience. After a concise survey of prior attempts—most of which rely on artificial “wait‑then‑read” probes or on stopping the system to take a snapshot—the authors argue that a proper consistency benchmark must capture the interplay among four dimensions: the storage system under test, the real‑world workload, the pattern of failures, and the consistency observed by clients.

To this end, they introduce a principled benchmarking framework centered on the notion of “observed consistency.” The framework consists of three tightly integrated components: (1) a lightweight proxy that intercepts every client request and automatically attaches logical timestamps and version vectors, (2) a failure‑logging module that records network partitions, node crashes, and resource saturation as they naturally occur, and (3) an analysis engine that reconstructs the happens‑before graph from the collected metadata and checks it against a chosen consistency model (e.g., read‑your‑writes, monotonic reads, causal consistency). Because the proxy only adds metadata and streams logs asynchronously, the overhead is measured at less than 3 % of the baseline latency, preserving the original workload characteristics.

The authors validate the approach on Apache Cassandra. They run twelve experiment configurations that combine three write consistency levels (ONE, QUORUM, ALL) with four read consistency levels, issuing 100 k read/write operations per configuration while a realistic failure pattern (random network partitions and node failures) is injected. The benchmark records, for each operation, the client‑visible version, the logical time, and the failure context. Analysis reveals that traditional “latency‑after‑write” methods dramatically under‑report staleness: the average client‑observed staleness is 250 ms (versus <100 ms reported by prior tools), with worst‑case delays exceeding 1.2 s. Moreover, even under QUORUM writes, consistency violations occur in about 5 % of reads during partition windows, contradicting the common belief that QUORUM guarantees strong enough consistency for most applications.

These findings have immediate implications for service‑level agreements (SLAs) and system design. Operators cannot rely solely on the configured consistency level; they must also consider the likelihood and duration of failures, and possibly adopt adaptive strategies such as dynamic consistency level adjustment or faster replica synchronization. The framework’s modular design makes it applicable beyond Cassandra—to systems like Riak, DynamoDB, CockroachDB, and emerging NewSQL databases—by simply swapping the proxy’s protocol handling.

In the discussion, the authors outline future extensions: automated trade‑off optimization that balances latency, throughput, and observed consistency; integration with machine‑learning‑based anomaly detection to flag unexpected consistency breaches in production; and support for multi‑region deployments where geographic latency adds another dimension to consistency behavior.

In conclusion, the paper delivers a comprehensive, minimally invasive methodology for measuring the real consistency guarantees of eventually‑consistent stores under realistic workloads and failure conditions. By unifying workload fidelity, failure realism, and precise client‑side observation, the proposed framework sets a new standard for consistency benchmarking and provides a solid foundation for both academic research and practical operations in the era of large‑scale NoSQL systems.