LogBase: A Scalable Log-structured Database System in the Cloud

Numerous applications such as financial transactions (e.g., stock trading) are write-heavy in nature. The shift from reads to writes in web applications has also been accelerating in recent years. Write-ahead-logging is a common approach for providing recovery capability while improving performance in most storage systems. However, the separation of log and application data incurs write overheads observed in write-heavy environments and hence adversely affects the write throughput and recovery time in the system. In this paper, we introduce LogBase - a scalable log-structured database system that adopts log-only storage for removing the write bottleneck and supporting fast system recovery. LogBase is designed to be dynamically deployed on commodity clusters to take advantage of elastic scaling property of cloud environments. LogBase provides in-memory multiversion indexes for supporting efficient access to data maintained in the log. LogBase also supports transactions that bundle read and write operations spanning across multiple records. We implemented the proposed system and compared it with HBase and a disk-based log-structured record-oriented system modeled after RAMCloud. The experimental results show that LogBase is able to provide sustained write throughput, efficient data access out of the cache, and effective system recovery.

💡 Research Summary

The paper introduces LogBase, a scalable log‑structured database designed for write‑heavy cloud applications. Traditional storage systems employ a write‑ahead‑log (WAL) plus separate data files, which incurs double I/O for each update and requires log replay during recovery, limiting write throughput and increasing recovery latency. LogBase eliminates this bottleneck by adopting a log‑only architecture: every write operation is appended to a single sequential log stored on a distributed file system (DFS) such as HDFS, and no separate data files are maintained. This design reduces disk I/O to a single sequential write per record, dramatically improving write bandwidth and lowering latency on commodity hardware.

To support efficient reads despite the log‑only layout, LogBase builds an in‑memory multiversion index for each tablet (partition). Each index entry contains the primary key, a commit timestamp (the logical order of the transaction), and a pointer to the log offset. Because the index resides in RAM, long‑tail queries that miss the cache still avoid costly index block reads, and the multiversion nature enables snapshot isolation and historical queries without additional overhead.

LogBase also provides full transactional semantics. It uses MVCC for concurrency control and a two‑phase commit protocol to guarantee atomicity across multiple records. Write operations generate new versions in the log; the index is updated atomically at commit time. Uncommitted or aborted writes are simply ignored during recovery, and obsolete versions are reclaimed by a background log‑compaction process.

The system is architected similarly to HBase: a set of tablet servers each manage a subset of tablets, and each tablet server maintains a single log instance for its tablets. The log is replicated across DFS nodes, ensuring durability against single‑node failures. Recovery after a node crash involves reconstructing the in‑memory indexes from the log, which is far faster than replaying separate data files.

LogBase further adopts vertical partitioning and column‑grouping to exploit column‑locality for analytical queries. Tables are split into column groups stored in separate physical partitions, allowing queries that touch only a few columns to avoid unnecessary I/O.

Experimental evaluation compares LogBase against HBase and a disk‑based log‑structured record system (LRS) modeled after RAMCloud. Using YCSB and a financial‑transaction benchmark on a 10‑node commodity cluster, LogBase achieves roughly double the sustained write throughput of HBase and 1.5× that of LRS, while maintaining comparable 99‑percentile read latencies (≈5 ms). In failure scenarios where multiple nodes crash simultaneously, LogBase restores service within a few seconds by rebuilding indexes, whereas LRS requires tens of seconds.

The authors discuss related work, highlighting that shadow‑paging (System R), delta‑record approaches (PostgreSQL), and pure WAL‑based systems all retain the log‑plus‑data separation that LogBase eliminates. LSM‑tree based systems (e.g., HBase, Cassandra) still rely on separate logs and thus cannot fully remove the write bottleneck.

Limitations are acknowledged: the in‑memory index may become too large for a tablet server with limited RAM, suggesting future work on index partitioning, disk‑based auxiliary indexes, or Bloom‑filter pruning. Log compaction overhead and its impact on write throughput also merit adaptive tuning. The paper proposes extending LogBase with SSD/NVMe tiers, hybrid memory‑disk architectures, and more sophisticated compaction policies to further improve performance and cost‑effectiveness.

Overall, LogBase demonstrates that a pure log‑only storage model, combined with multiversion in‑memory indexing and lightweight transaction management, can deliver high write scalability, fast recovery, and acceptable read performance for modern cloud‑native, write‑intensive applications.