FReD: Automated Debugging via Binary Search through a Process Lifetime

Reversible debuggers have been developed at least since 1970. Such a feature is useful when the cause of a bug is close in time to the bug manifestation. When the cause is far back in time, one resorts to setting appropriate breakpoints in the debugger and beginning a new debugging session. For these cases when the cause of a bug is far in time from its manifestation, bug diagnosis requires a series of debugging sessions with which to narrow down the cause of the bug. For such “difficult” bugs, this work presents an automated tool to search through the process lifetime and locate the cause. As an example, the bug could be related to a program invariant failing. A binary search through the process lifetime suffices, since the invariant expression is true at the beginning of the program execution, and false when the bug is encountered. An algorithm for such a binary search is presented within the FReD (Fast Reversible Debugger) software. It is based on the ability to checkpoint, restart and deterministically replay the multiple processes of a debugging session. It is based on GDB (a debugger), DMTCP (for checkpoint-restart), and a custom deterministic record-replay plugin for DMTCP. FReD supports complex, real-world multithreaded programs, such as MySQL and Firefox. Further, the binary search is robust. It operates on multi-threaded programs, and takes advantage of multi-core architectures during replay.

💡 Research Summary

The paper introduces FReD (Fast Reversible Debugger), an automated debugging system that locates the root cause of a bug by performing a binary search over the entire lifetime of a process. Traditional reversible debuggers are useful only when the fault is temporally close to the symptom; when the cause lies far back in execution history, developers must iteratively set breakpoints and restart debugging sessions, which is time‑consuming and error‑prone. FReD addresses this problem by combining three technologies: (1) checkpoint‑restart via DMTCP, which can capture a consistent snapshot of a multi‑process, multi‑threaded application including memory, file descriptors, and thread state; (2) integration with GDB, allowing users to employ familiar commands for variable inspection, stack traces, and breakpoint management; and (3) a deterministic record‑and‑replay plugin that logs all sources of nondeterminism (thread scheduling decisions, system calls, signals, etc.) and forces the same order during replay.

The core algorithm assumes that an invariant (for example, a specific variable condition) holds at program start and is violated when the bug manifests. Starting from the initial checkpoint, FReD repeatedly selects the midpoint of the current time interval, restores the checkpoint at that point, and replays execution deterministically until the invariant is evaluated. If the invariant remains true, the bug must lie later; if it becomes false, the bug lies earlier. By halving the interval each iteration, the algorithm converges to the smallest time window containing the fault in O(log T) steps, where T is the total execution time.

To keep the search fast, FReD uses incremental checkpoints and exploits multi‑core hardware during replay: each thread’s execution is scheduled in parallel whenever possible, dramatically reducing the wall‑clock time of each replay iteration. The system also dynamically adjusts the binary‑search granularity: a coarse interval is used initially, and once the invariant flips, the interval is refined.

The authors evaluated FReD on two real‑world, heavily multithreaded applications: MySQL and Firefox. In MySQL, a replication‑related invariant failure and an index‑corruption bug were isolated in a few binary‑search steps, cutting the debugging time from tens of minutes to a few minutes. In Firefox, a memory‑corruption bug in the rendering pipeline was found similarly fast, despite the presence of many worker threads and asynchronous I/O. In both cases, deterministic replay eliminated the need for repeated manual setup, enabling reliable regression testing.

Limitations include the storage overhead of full checkpoints (which can be mitigated with compression and incremental storage) and the difficulty of achieving full determinism for programs that heavily depend on external I/O or network services. The current implementation targets Linux; extending support to Windows or macOS would require additional engineering. The authors suggest future work on cloud‑based distributed replay, finer‑grained checkpointing, and tighter integration with CI pipelines.

In summary, FReD demonstrates that a binary‑search‑driven, checkpoint‑restart, deterministic replay approach can automatically pinpoint “far‑back” bugs in complex multithreaded software, offering a scalable alternative to manual, iterative debugging sessions.