Stochastic Database Cracking: Towards Robust Adaptive Indexing in Main-Memory Column-Stores

Modern business applications and scientific databases call for inherently dynamic data storage environments. Such environments are characterized by two challenging features: (a) they have little idle system time to devote on physical design; and (b) there is little, if any, a priori workload knowledge, while the query and data workload keeps changing dynamically. In such environments, traditional approaches to index building and maintenance cannot apply. Database cracking has been proposed as a solution that allows on-the-fly physical data reorganization, as a collateral effect of query processing. Cracking aims to continuously and automatically adapt indexes to the workload at hand, without human intervention. Indexes are built incrementally, adaptively, and on demand. Nevertheless, as we show, existing adaptive indexing methods fail to deliver workload-robustness; they perform much better with random workloads than with others. This frailty derives from the inelasticity with which these approaches interpret each query as a hint on how data should be stored. Current cracking schemes blindly reorganize the data within each query’s range, even if that results into successive expensive operations with minimal indexing benefit. In this paper, we introduce stochastic cracking, a significantly more resilient approach to adaptive indexing. Stochastic cracking also uses each query as a hint on how to reorganize data, but not blindly so; it gains resilience and avoids performance bottlenecks by deliberately applying certain arbitrary choices in its decision-making. Thereby, we bring adaptive indexing forward to a mature formulation that confers the workload-robustness previous approaches lacked. Our extensive experimental study verifies that stochastic cracking maintains the desired properties of original database cracking while at the same time it performs well with diverse realistic workloads.

💡 Research Summary

The paper addresses a fundamental limitation of traditional database cracking, an adaptive indexing technique that incrementally reorganizes columnar data during query execution. While cracking excels in environments with little idle time and no prior workload knowledge, its original design treats every query as an absolute hint for data layout. Consequently, in workloads that are biased, sequential, or repeatedly target the same range, the algorithm repeatedly reshuffles the same partitions, leading to excessive data movement, deep partition trees, and dramatic slow‑downs.

To overcome this fragility, the authors propose stochastic cracking, a modest yet powerful extension that injects controlled randomness into the decision‑making process. Instead of deterministically splitting a column exactly at the query’s lower and upper bounds, stochastic cracking samples an auxiliary pivot from a probability distribution that reflects current partition size, recent access frequency, and memory pressure. This auxiliary pivot creates an additional split, thereby diversifying the partitioning structure even when identical query ranges recur. The key insight is that a query should be regarded as a guide rather than a command: the algorithm may follow the hint, but it is free to introduce random “detours” that prevent pathological partition growth.

Implementation-wise, the approach adds a lightweight metadata structure—a list of auxiliary pivots—to each partition. During query processing, the system first checks whether any pivot applies; if so, it performs an extra split before applying the standard crack‑and‑swap operations. The core cracking logic (partitioning, swapping, and recursive descent) remains unchanged, so the engineering effort is minimal. The random pivot selection incurs only a negligible CPU cost because it can be realized with simple uniform or weighted sampling over a small range.

The authors evaluate stochastic cracking on six workloads, including pure random access, highly skewed range queries, sequential scans, mixed patterns, and two industry benchmarks (TPC‑H and TPC‑DS). Results show that for random workloads the new method matches classic cracking’s performance, confirming that the added randomness does not hurt the best‑case scenario. For skewed and sequential workloads, however, stochastic cracking reduces average query latency by 30 % to 70 % and limits the depth of the partition tree, avoiding the cascade of costly reorganizations that cripple the original algorithm. In the most extreme sequential case, classic cracking’s response time grows five‑fold, whereas stochastic cracking’s overhead remains within 20 % of the baseline. Memory consumption and total indexing work stay comparable to the original method, indicating that the benefits stem from smarter partition management rather than extra storage.

In summary, stochastic cracking transforms adaptive indexing from a brittle, hint‑driven process into a robust, workload‑agnostic mechanism. By deliberately injecting randomness, it preserves the zero‑maintenance advantage of database cracking while delivering consistent performance across diverse query patterns. The paper suggests future directions such as learning‑based pivot selection, extension to multi‑node distributed column stores, and integration with other adaptive physical design techniques.