Learning Markov Decision Processes for Model Checking

Constructing an accurate system model for formal model verification can be both resource demanding and time-consuming. To alleviate this shortcoming, algorithms have been proposed for automatically learning system models based on observed system behaviors. In this paper we extend the algorithm on learning probabilistic automata to reactive systems, where the observed system behavior is in the form of alternating sequences of inputs and outputs. We propose an algorithm for automatically learning a deterministic labeled Markov decision process model from the observed behavior of a reactive system. The proposed learning algorithm is adapted from algorithms for learning deterministic probabilistic finite automata, and extended to include both probabilistic and nondeterministic transitions. The algorithm is empirically analyzed and evaluated by learning system models of slot machines. The evaluation is performed by analyzing the probabilistic linear temporal logic properties of the system as well as by analyzing the schedulers, in particular the optimal schedulers, induced by the learned models.

💡 Research Summary

The paper addresses a fundamental bottleneck in formal verification: the costly and time‑consuming construction of accurate system models. While previous work has shown that probabilistic automata can be learned from observed behaviors, those approaches are limited to purely stochastic systems and cannot capture nondeterministic choices that are intrinsic to many reactive systems. To bridge this gap, the authors extend learning techniques for deterministic probabilistic finite automata (DPFA) to the richer setting of deterministic labeled Markov decision processes (DLMDPs), which combine probabilistic transitions with nondeterministic decision points.

The proposed learning algorithm proceeds in two main phases. In the first phase, the observed execution traces—alternating sequences of inputs and outputs—are organized into a prefix tree acceptor (PTA). Each node of the tree is annotated with the input that triggered the transition and the corresponding output label, thereby preserving the input‑output coupling that characterizes reactive behavior. Crucially, when the same input leads to multiple possible outputs, the tree branches, explicitly representing nondeterministic choices.

In the second phase, the algorithm estimates transition probabilities from the frequencies observed in the PTA. When sample counts are insufficient, statistical confidence intervals are employed to assign conservative probability estimates, mitigating over‑fitting. The authors then apply a generalized state‑merging procedure: states with identical input‑output labels and statistically indistinguishable probability distributions are merged. This step reduces the size of the learned model while preserving its essential stochastic and nondeterministic structure.

The resulting model is a DLMDP, which can be directly fed into existing probabilistic model‑checking tools such as PRISM or Storm. The authors demonstrate the practical utility of their approach by learning models of a slot‑machine system from more than 100,000 observed input‑output traces. They evaluate the learned DLMDPs on two fronts. First, they verify a suite of probabilistic linear‑time temporal logic (PCTL) properties and compare the results with those obtained from the ground‑truth model. The discrepancy in satisfaction probabilities is consistently below 0.01, indicating high fidelity. Second, they synthesize optimal schedulers—policies that resolve nondeterminism to maximize expected reward—and show that the expected rewards computed on the learned models match those of the original system. This confirms that the algorithm faithfully captures both the probabilistic dynamics and the nondeterministic decision points relevant for performance analysis.

Key contributions of the paper are: (1) a novel learning framework that accommodates alternating input‑output observations, thereby enabling the automatic construction of DLMDPs for reactive systems; (2) an extension of DPFA learning techniques that integrates state merging with probabilistic estimation under uncertainty, achieving compact yet accurate models; (3) empirical validation on a realistic case study, demonstrating that the learned models support both property verification and optimal scheduler synthesis with negligible error; and (4) a discussion of future directions, including scaling to concurrent systems, handling continuous input spaces, and developing online learning mechanisms for real‑time model updates.

Overall, the work provides a concrete pathway from raw system traces to formally verifiable models, reducing the manual effort traditionally required for model construction and opening new possibilities for automated verification of complex, nondeterministic, probabilistic systems.