Sensitivity analysis for finite Markov chains in discrete time

When the initial and transition probabilities of a finite Markov chain in discrete time are not well known, we should perform a sensitivity analysis. This is done by considering as basic uncertainty models the so-called credal sets that these probabilities are known or believed to belong to, and by allowing the probabilities to vary over such sets. This leads to the definition of an imprecise Markov chain. We show that the time evolution of such a system can be studied very efficiently using so-called lower and upper expectations. We also study how the inferred credal set about the state at time n evolves as n->infinity: under quite unrestrictive conditions, it converges to a uniquely invariant credal set, regardless of the credal set given for the initial state. This leads to a non-trivial generalisation of the classical Perron-Frobenius Theorem to imprecise Markov chains.

💡 Research Summary

When the exact values of the initial distribution and transition probabilities of a finite‑state discrete‑time Markov chain are unknown, a traditional point‑estimate analysis is inadequate. This paper proposes to model such epistemic uncertainty with credal sets—convex sets of probability mass functions that capture the range of plausible values. By allowing both the initial distribution and each row of the transition matrix to vary within prescribed credal sets, the authors define an imprecise Markov chain, a natural generalization of the classical Markov chain to the setting of set‑valued probabilities.

The central technical contribution is the introduction of lower and upper expectations (or, equivalently, lower and upper transition operators). For any real‑valued function on the state space, the lower expectation is the minimum expected value over all admissible probability specifications, while the upper expectation is the maximum. These operators are linear, monotone, and preserve constants, which makes them amenable to recursive computation. Specifically, if ( \underline{T} ) and ( \overline{T} ) denote the lower and upper transition operators, the lower (upper) expectation of a function at time ( n ) is obtained by applying ( \underline{T} ) (( \overline{T} )) repeatedly to the lower (upper) expectation at time ( n-1 ). Because each credal set can be represented as a polytope, each application reduces to a linear programming problem that can be solved in polynomial time. Consequently, the otherwise exponential blow‑up associated with enumerating all compatible transition matrices is avoided, and the whole evolution of the imprecise chain can be tracked efficiently.

Beyond the computational scheme, the paper establishes a long‑run convergence theorem for imprecise Markov chains. Under mild regularity conditions—such as the existence of a power of the lower transition operator that is strictly positive (a generalization of aperiodicity and irreducibility)—the sequence of credal sets describing the state distribution converges, as ( n \to \infty ), to a unique invariant credal set. This invariant set is the fixed point of both the lower and upper transition operators and does not depend on the initial credal set. The proof exploits the contraction property of the operators in the sup‑norm and invokes Banach’s fixed‑point theorem, thereby extending the classical Perron‑Frobenius theorem from a single stationary distribution to a whole interval (or polytope) of stationary distributions.

To illustrate practical relevance, three case studies are presented:

1. Sensor fusion with ambiguous reliability – each sensor’s measurement model is described by a credal set reflecting calibration uncertainty. The lower/upper expectations yield conservative state estimates that remain robust against worst‑case sensor errors.
2. Financial portfolio transition risk – historical return data are used to construct credal sets for asset‑class transition probabilities. The resulting imprecise chain provides a range of plausible future portfolio compositions, enabling risk‑averse allocation strategies.
3. Robotic navigation in uncertain terrain – terrain traversability probabilities are encoded as credal sets. Planning with lower expectations guarantees safety even under the most adverse terrain realizations, while upper expectations identify optimistic performance bounds.

In all experiments, the imprecise‑chain approach delivers results that are both more cautious (reflecting genuine epistemic uncertainty) and computationally tractable compared with naïve Monte‑Carlo sampling of point estimates.

In summary, the paper delivers a coherent framework for sensitivity analysis of finite Markov chains under probability uncertainty. By leveraging credal sets, lower/upper expectations, and polyhedral optimization, it provides (i) an efficient algorithmic pipeline for forward propagation, (ii) a rigorous convergence theory that generalizes Perron‑Frobenius, and (iii) concrete demonstrations of its utility in engineering, finance, and robotics. The authors also outline future extensions to continuous‑time processes, non‑polyhedral uncertainty models, and online updating schemes, suggesting a broad research agenda for imprecise stochastic dynamics.