Deterministic execution offers many benefits for debugging, fault tolerance, and security. Running parallel programs deterministically is usually difficult and costly, however - especially if we desire system-enforced determinism, ensuring precise repeatability of arbitrarily buggy or malicious software. Determinator is a novel operating system that enforces determinism on both multithreaded and multi-process computations. Determinator's kernel provides only single-threaded, "shared-nothing" address spaces interacting via deterministic synchronization. An untrusted user-level runtime uses distributed computing techniques to emulate familiar abstractions such as Unix processes, file systems, and shared memory multithreading. The system runs parallel applications deterministically both on multicore PCs and across nodes in a cluster. Coarse-grained parallel benchmarks perform and scale comparably to - sometimes better than - conventional systems, though determinism is costly for fine-grained parallel applications.
Deep Dive into Efficient System-Enforced Deterministic Parallelism.
Deterministic execution offers many benefits for debugging, fault tolerance, and security. Running parallel programs deterministically is usually difficult and costly, however - especially if we desire system-enforced determinism, ensuring precise repeatability of arbitrarily buggy or malicious software. Determinator is a novel operating system that enforces determinism on both multithreaded and multi-process computations. Determinator’s kernel provides only single-threaded, “shared-nothing” address spaces interacting via deterministic synchronization. An untrusted user-level runtime uses distributed computing techniques to emulate familiar abstractions such as Unix processes, file systems, and shared memory multithreading. The system runs parallel applications deterministically both on multicore PCs and across nodes in a cluster. Coarse-grained parallel benchmarks perform and scale comparably to - sometimes better than - conventional systems, though determinism is costly for fine-grai
It is often useful to run software deterministically, ensuring a given program and input always yields exactly the same result. Deterministic execution makes bugs reproducible, and is required for "record-and-replay" debugging [28,40]. Fault tolerance [15,18,49] and accountability mechanisms [33] rely on execution being deterministic and bit-for-bit identical across state replicas. Intrusion analysis [23,36] and timing channel control [4] can further benefit from system-enforced determinism, where the system prevents application code from depending on execution timing or other unintended inputs even if the code is maliciously designed to do so.
Multicore processors and ubiquitous parallelism make programming environments increasingly nondeterministic, however. Nondeterminism makes software harder to develop and debug [43,44]. Race detectors help [27,45], but even properly synchronized programs may have higher-level heisenbugs [3]. The cost of logging and replaying the internal nondeterministic events in parallel software [20,24] can be orders of magnitude higher than that of logging only a computation’s external inputs, especially for system-enforced replay [23,24]. This cost usually precludes logging “normal-case” execution, di-minishing the technique’s effectiveness. A heisenbug or intrusion that manifests “in the field” with logging disabled may not reappear during subsequent logged attempts to reproduce it-especially with malware designed to evade analysis by detecting the timing impact of logging or virtualization [30].
Motivated by its many uses, we would like systemenforced determinism to be available for normal-case execution of parallel applications. To test this goal’s feasibility, we built Determinator, an operating system that not only executes individual processes deterministically, as in deterministic user-level scheduling [8,9], but can enforce determinism on hierarchies of interacting processes. Rerunning a multi-process Determinator computation with the same inputs yields exactly the same outputs, without internal event logging. Determinator treats all potential nondeterministic inputs to a computationincluding all timing information-as “privileged information,” which normal applications cannot obtain except via controlled channels. We treat deterministic execution as not just a debugging tool but a security principle: if malware infects an unprivileged Determinator application, it should be unable to evade replay-based analysis.
System-enforced determinism is challenging because current programming environments and APIs are riddled with timing dependencies. Most shared-memory parallel code uses mutual exclusion primitives: even when used correctly, timing determines the application-visible order in which competing threads acquire a mutex. Concurrency makes names allocated from shared namespaces, such as pointers returned by malloc() and file descriptors returned by open(), timing-dependent. Synchronizing operations like semaphores, message queues, and wait() nondeterministically return “the first” event, message, or terminated process available. Even singlethreaded processes are nondeterministic when run in parallel, due to their interleaved accesses to shared resources. A parallel ‘make -j’ command often presents a chaotic mix of its child tasks’ outputs, for example, and missing dependencies can yield “makefile heisenbugs” that manifest only under parallel execution.
Addressing these challenges in Determinator led us to the insight that timing dependencies commonly fall into a few categories: unintended interactions via shared state or namespaces; synchronization abstractions with share-able endpoints; true dependencies on “real-world” time; and application-level scheduling. Determinator avoids physically shared state by isolating concurrent activities during normal execution, allowing interaction only at explicit synchronization points. The kernel’s API uses local, application-chosen names in place of shared, OSmanaged namespaces. Synchronization primitives operate “one-to-one,” between specific threads, preventing threads from “racing” to an operation. Determinator treats access to real-world time as I/O, controlling it as with other devices such as disk or network. Finally, Determinator requires scheduling to be separated from application logic and handled by the system, or else emulated using a deterministic, virtual notion of “time.”
Since we wish to derive basic principles for systemenforced determinism, Determinator currently makes no attempt at compatibility with existing operating systems, and provides limited compatibility with existing APIs. The kernel’s low-level API offers only one user-visible abstraction, spaces, representing execution state and virtual memory, and only three system calls by which spaces synchronize and communicate. The API’s minimality facilitates both experimentation and reasoning about its determinism. Despites this simplicity, our untrusted, user-
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
