Characterization and computation of infinite horizon specifications over Markov processes

This work is devoted to the formal verification of specifications over general discrete-time Markov processes, with an emphasis on infinite-horizon properties. These properties, formulated in a modal logic known as PCTL, can be expressed through value functions defined over the state space of the process. The main goal is to understand how structural features of the model (primarily the presence of absorbing sets) influence the uniqueness of the solutions of corresponding Bellman equations. Furthermore, this contribution shows that the investigation of these structural features leads to new computational techniques to calculate the specifications of interest: the emphasis is to derive approximation techniques with associated explicit convergence rates and formal error bounds.

💡 Research Summary

The paper addresses the formal verification of infinite‑horizon specifications expressed in Probabilistic Computation Tree Logic (PCTL) over general discrete‑time Markov processes (DTMPs). Such specifications—e.g., “always”, “repeatedly”, or “eventually with probability 1”—are captured by value functions that assign to each state the probability that the specification will be satisfied when the process starts from that state. These value functions are fixed points of Bellman equations, and the central theoretical question is under what conditions these equations admit a unique solution.

The authors first conduct a structural analysis of the underlying Markov model, focusing on the presence of absorbing sets—subsets of states that, once entered, cannot be left. They prove that if the DTMP contains a non‑trivial absorbing set and the transition kernel is such that the process remains in the set with probability one, then the associated Bellman operator is contractive on the complement of the set and the value function has a unique minimal (and maximal) solution. Conversely, when no absorbing set exists or when the state space decomposes into several mutually non‑communicating absorbing classes, the Bellman operator may have multiple fixed points, leading to non‑uniqueness of the solution. This insight provides a clear, model‑based criterion for deciding whether standard value‑iteration or linear‑programming approaches will converge to a meaningful result.

Building on this structural insight, the paper proposes a new computational framework. The state space is discretized either by a uniform grid or by a sampling‑based approximation, yielding an approximate transition operator. Two sources of error are identified: (1) the bias introduced by approximating the true kernel, and (2) the residual error after a finite number of Bellman iterations. Under mild regularity assumptions (Lipschitz continuity of the kernel), the kernel approximation error scales linearly with the discretization granularity. Moreover, because the Bellman operator is shown to be γ‑contractive (γ < 1) on the relevant subspace, the iteration error decays geometrically as γⁿ. By combining these two analyses, the authors derive an explicit error bound of the form

  ‖V̂ − V‖ ≤ (γᵏ/(1−γ))·ε₁ + ε₂,

where ε₁ is the kernel‑approximation error, ε₂ is the stopping tolerance, and k is the number of iterations. This bound enables the automatic selection of discretization resolution and iteration count to guarantee a user‑specified precision ε.

The practical relevance of the approach is demonstrated on two benchmark families. In the first, standard Bayesian network models are examined for infinite‑horizon reachability probabilities; in the second, a mobile robot navigation problem is analyzed for perpetual safety (avoidance of hazardous regions). In cases where absorbing sets are clearly identifiable, the computed probabilities match analytically derived values within the theoretically predicted error margins. When absorbing sets are ambiguous or absent, the algorithm detects the lack of contractivity, issues a warning, and suggests model refinement (e.g., redefining absorbing regions).

Compared with prior work, which typically relies on value iteration without rigorous guarantees of uniqueness or convergence rate, this paper contributes (i) a precise characterization of when unique solutions exist based on absorbing‑set structure, (ii) a provably convergent approximation scheme with explicit convergence rates, and (iii) formal error bounds that can be incorporated into automated verification tools. These contributions advance the state of the art in the verification of stochastic systems, particularly for safety‑critical applications where infinite‑horizon guarantees and quantitative error certificates are indispensable.