Efficient System-Enforced Deterministic Parallelism

Deterministic execution offers many benefits for debugging, fault tolerance, and security. Running parallel programs deterministically is usually difficult and costly, however - especially if we desire system-enforced determinism, ensuring precise repeatability of arbitrarily buggy or malicious software. Determinator is a novel operating system that enforces determinism on both multithreaded and multi-process computations. Determinator’s kernel provides only single-threaded, “shared-nothing” address spaces interacting via deterministic synchronization. An untrusted user-level runtime uses distributed computing techniques to emulate familiar abstractions such as Unix processes, file systems, and shared memory multithreading. The system runs parallel applications deterministically both on multicore PCs and across nodes in a cluster. Coarse-grained parallel benchmarks perform and scale comparably to - sometimes better than - conventional systems, though determinism is costly for fine-grained parallel applications.

💡 Research Summary

The paper presents Determinator, a novel operating system that enforces deterministic execution for both multithreaded and multiprocess computations. Deterministic execution—where the same program and inputs always produce identical outputs—is valuable for debugging, fault‑tolerant replication, and security, yet achieving it in a parallel setting is notoriously difficult because modern APIs and hardware expose timing‑dependent nondeterminism (e.g., mutex acquisition order, file descriptor allocation, real‑time clocks). Determinator tackles this by redesigning the kernel from the ground up, providing only “shared‑nothing” single‑threaded address spaces called spaces and a minimal set of three system calls: Put, Get, and Ret.

Design principles guide the kernel: (1) isolate all mutable state between explicit synchronization points; (2) replace global names (PIDs, file descriptors, mmap addresses) with application‑chosen local identifiers; (3) require that every synchronization operation specify both the target and the exact point in the target’s execution; (4) treat all sources of real‑time or other nondeterministic input as I/O devices, accessible only through controlled channels; and (5) separate scheduling decisions from application logic. By obeying these rules, the kernel eliminates the classic sources of timing‑dependent nondeterminism.

A space contains a private register set and a private virtual memory region, analogous to a single‑threaded Unix process but without any shared kernel resources. The three system calls implement a rendezvous protocol: a parent can Put a child (creating it if necessary), copy registers and memory, set permissions, optionally snapshot the child’s address space, and start execution. The child runs until it executes Ret, which stops it and returns control (and a status code) to the parent. The parent may then Get the child’s state, copying registers, memory, or performing a Merge that copies only the bytes that changed since the last snapshot. Merge also detects write‑write conflicts, treating them as programming errors. Because each interaction is a blocking, one‑to‑one exchange, the whole hierarchy of spaces can be modeled as a Kahn process network, which is known to be deterministic.

Since the kernel offers only these low‑level primitives, a user‑level runtime is built on top to provide familiar abstractions: a logically shared file system (implemented via file replication and versioning), distributed shared memory (using copy‑on‑write pages and Merge to emulate a single address space), and a deterministic thread library (splitting execution into quanta of instruction counts rather than wall‑clock time). The runtime also mediates all I/O: only the root space can directly access disks, networks, or clocks; all other spaces must request such services through their ancestors, allowing the system to control nondeterministic inputs.

The prototype runs on multicore x86‑64 machines and on a four‑node cluster. Experiments with coarse‑grained benchmarks—parallel make, matrix multiplication, FFT—show that Determinator’s performance and scalability are comparable to, and sometimes better than, conventional nondeterministic systems. The deterministic guarantees come essentially for free because the benchmarks involve relatively infrequent synchronization and large computational kernels between sync points. In contrast, fine‑grained microbenchmarks that heavily use locks, barriers, or frequent small updates suffer substantial overhead due to the cost of page copying and merging; speed‑ups are typically an order of magnitude lower than native execution. I/O‑bound workloads also expose a bottleneck: the root space becomes a serialization point for all device accesses, a limitation acknowledged by the authors.

The authors position Determinator against prior work such as deterministic schedulers, record‑and‑replay systems, and deterministic sandbox VMs. Those approaches usually sit atop existing OSes and rely on extensive logging, which incurs high runtime and storage costs and can be subverted by malicious timing attacks. By moving determinism enforcement into the kernel itself, Determinator provides a stronger, more transparent guarantee that even buggy or malicious user code cannot escape replay‑based analysis.

In summary, the paper makes three contributions: (1) a set of OS‑level design principles for system‑enforced determinism; (2) a concrete kernel API that embodies those principles; and (3) a full user‑space runtime that reconstructs familiar Unix abstractions while preserving deterministic execution, demonstrated to be practical for coarse‑grained parallel applications. Limitations include poor performance on fine‑grained parallelism, lack of persistent storage, a restrictive space hierarchy, and I/O bottlenecks. Future work suggested includes integrating deterministic sandboxes into mainstream OSes, optimizing page‑level merging (perhaps with hardware support), and extending the I/O model to reduce the root‑space bottleneck. Overall, Determinator offers a compelling proof‑of‑concept that deterministic execution can be made a first‑class OS service without prohibitive overhead for many real‑world workloads.