Object Database Benchmarks

The need for performance measurement tools appeared soon after the emergence of the first Object-Oriented Database Management Systems (OODBMSs), and proved important for both designers and users (Atkinson & Maier, 1990). Performance evaluation is useful to designers to determine elements of architecture and more generally to validate or refute hypotheses regarding the actual behavior of an OODBMS. Thus, performance evaluation is an essential component in the development process of well-designed and efficient systems. Users may also employ performance evaluation, either to compare the efficiency of different technologies before selecting an OODBMS or to tune a system.Performance evaluation by experimentation on a real system is generally referred to as benchmarking. It consists in performing a series of tests on a given OODBMS to estimate its performance in a given setting. Benchmarks are generally used to compare the global performance of OODBMSs, but they can also be exploited to illustrate the advantages of one system or another in a given situation, or to determine an optimal hardware configuration. Typically, a benchmark is constituted of two main elements: a workload model constituted of a database and a set of read and write operations to apply on this database, and a set of performance metrics.

💡 Research Summary

The paper provides a comprehensive treatment of performance benchmarking for Object‑Oriented Database Management Systems (OODBMSs). It begins by noting that, unlike relational databases, OODBMSs store complex objects, support inheritance, polymorphism, and deep reference graphs, which makes traditional relational benchmarks unsuitable. Consequently, a dedicated benchmark must model both the data layout (schema, object graph topology, object size distribution) and the workload (object navigation, association queries, insert/delete/update of composite objects, versioning, and transaction management).

A benchmark is defined as the combination of a workload model and a set of performance metrics. The workload model is parameterized to reflect real‑world domains such as CAD, GIS, multimedia, or e‑commerce, allowing researchers to vary graph depth, fan‑out, and reference path length. The operation set includes read‑only traversals, multi‑object fetches, complex updates, and transaction‑level operations that stress caching, indexing, and concurrency control mechanisms.

Performance metrics go beyond average response time; they encompass percentile latencies (e.g., 95th percentile), throughput (transactions per second), CPU, I/O, memory consumption, network bandwidth, and scalability indicators (performance change when adding nodes). The authors stress rigorous experimental design: repeated runs (typically ≥30), statistical significance testing, and strict control of hardware, OS, and network settings to ensure reproducibility.

From a designer’s perspective, benchmark results guide architectural choices such as object cache size, page size, index structure (B‑tree vs. R‑tree), and replication or sharding strategies. For example, increasing cache size may cut navigation latency by 30 % while raising memory usage, allowing a trade‑off analysis. From a user’s perspective, the same benchmark framework enables side‑by‑side comparison of competing OODBMS products under identical conditions, providing concrete evidence for procurement decisions. It also supports hardware tuning: measuring performance differences between SSD and HDD storage, varying memory capacity, or adjusting network bandwidth to identify the optimal configuration.

Finally, the paper calls for standardization and automation of OODBMS benchmarks. Currently, each research group creates its own benchmark, hindering cross‑study comparison. Establishing a common set of workloads and operations, together with automated execution and analysis pipelines, would foster a shared performance database for the OODBMS community and accelerate both research and commercial adoption.