Efficient Large-scale Trace Checking Using MapReduce

The problem of checking a logged event trace against a temporal logic specification arises in many practical cases. Unfortunately, known algorithms for an expressive logic like MTL (Metric Temporal Logic) do not scale with respect to two crucial dimensions: the length of the trace and the size of the time interval for which logged events must be buffered to check satisfaction of the specification. The former issue can be addressed by distributed and parallel trace checking algorithms that can take advantage of modern cloud computing and programming frameworks like MapReduce. Still, the latter issue remains open with current state-of-the-art approaches. In this paper we address this memory scalability issue by proposing a new semantics for MTL, called lazy semantics. This semantics can evaluate temporal formulae and boolean combinations of temporal-only formulae at any arbitrary time instant. We prove that lazy semantics is more expressive than standard point-based semantics and that it can be used as a basis for a correct parametric decomposition of any MTL formula into an equivalent one with smaller, bounded time intervals. We use lazy semantics to extend our previous distributed trace checking algorithm for MTL. We evaluate the proposed algorithm in terms of memory scalability and time/memory tradeoffs.

💡 Research Summary

The paper tackles two intertwined scalability challenges that arise when verifying execution traces against Metric Temporal Logic (MTL) specifications in modern, data‑intensive systems. The first challenge is the sheer size of logs—often hundreds of gigabytes—requiring massive parallel computation. The second, more subtle challenge, is the memory consumption caused by large time intervals embedded in MTL operators (e.g., “eventually within a week”). Traditional point‑based semantics forces a verifier to buffer all events that fall inside such intervals, leading to buffer sizes that grow linearly with the interval length and can quickly exceed the memory available on a single node, even in a distributed setting.

To address the memory issue, the authors introduce lazy semantics, a new interpretation of MTL. Under lazy semantics, atomic propositions are still evaluated only at timestamps present in the trace, but temporal operators (Until, Eventually, etc.) can be evaluated at any arbitrary time instant without requiring the full set of intervening events to be materialized in memory. This decouples the evaluation of temporal constraints from the need to keep the entire interval in a buffer. The authors prove that lazy semantics is strictly more expressive than the classic point‑based semantics: every point‑based formula can be rewritten into an equivalent lazy‑semantics formula, while there exist lazy‑semantics formulas that have no point‑based counterpart.

The key technical contribution built on lazy semantics is a parametric decomposition of arbitrary MTL formulas. Given a user‑chosen constant (K), the decomposition rewrites any formula (\varphi) into an equivalent formula (\varphi’) where every temporal operator’s upper bound is at most (K). This transformation preserves logical equivalence and is sound for both point‑based and lazy semantics. Consequently, a verifier only ever needs to buffer at most (K) time units of events, regardless of the original interval sizes, dramatically reducing the memory footprint.

The authors integrate these ideas into a distributed trace‑checking algorithm that runs on the MapReduce paradigm, specifically implemented with Apache Spark. In the Map phase, each log entry (a timestamp together with the set of atomic events) is emitted as a key‑value pair. The Reduce phase performs the lazy‑semantics evaluation, applying the parametric decomposition on the fly. Because the decomposition guarantees a bounded upper interval, each reducer’s local buffer never exceeds (K) events, making the algorithm’s memory usage independent of trace length and original interval magnitudes.

Experimental evaluation uses a real‑world Wikipedia page‑traffic dataset (≈320 GB) and a suite of MTL specifications with varying interval sizes. The authors vary the decomposition constant (K) (e.g., 10 s, 30 s, 60 s) and measure memory consumption, total execution time, and the trade‑off between them. Results show a clear space‑time trade‑off: smaller (K) yields lower memory usage (as low as 2 GB) but incurs modestly higher runtime due to increased number of reduction steps; larger (K) reduces runtime at the cost of higher memory. Compared against state‑of‑the‑art tools such as MonPoly and earlier MTL‑MapReduce implementations, the proposed approach succeeds in cases where those tools run out of memory, while achieving comparable or better execution times.

In summary, the paper’s contributions are:

1. Lazy semantics for MTL, proved to be strictly more expressive than point‑based semantics.
2. Parametric decomposition that bounds all temporal intervals by a user‑defined constant while preserving equivalence.
3. A scalable MapReduce‑based trace‑checking algorithm that leverages lazy semantics and the decomposition to achieve memory‑bounded verification.
4. Comprehensive empirical validation demonstrating substantial memory savings and practical feasibility on cloud‑scale infrastructures.

The work opens the door for efficient, memory‑constrained verification of complex temporal properties in large‑scale distributed systems, with potential applications in runtime monitoring, security log analysis, and compliance checking across cloud platforms.