Memoryless Policy Iteration for Episodic POMDPs

Memoryless and finite-memory policies offer a practical alternative for solving partially observable Markov decision processes (POMDPs), as they operate directly in the output space rather than in the high-dimensional belief space. However, extending classical methods such as policy iteration to this setting remains difficult; the output process is non-Markovian, making policy-improvement steps interdependent across stages. We introduce a new family of monotonically improving policy-iteration algorithms that alternate between single-stage output-based policy improvements and policy evaluations according to a prescribed periodic pattern. We show that this family admits optimal patterns that maximize a natural computational-efficiency index, and we identify the simplest pattern with minimal period. Building on this structure, we further develop a model-free variant that estimates values from data and learns memoryless policies directly. Across several POMDPs examples, our method achieves significant computational speedups over policy-gradient baselines and recent specialized algorithms in both model-based and model-free settings.

💡 Research Summary

The paper tackles the long‑standing challenge of solving partially observable Markov decision processes (POMDPs) with policies that depend only on the current observation—a class known as memoryless (or finite‑window) policies. While such policies avoid the exponential blow‑up of belief‑space methods, the output process becomes non‑Markovian, breaking the core assumption of classical policy‑iteration (PI) that each stage can be improved independently.

To overcome this, the authors introduce a novel family of policy‑iteration algorithms that interleave single‑stage, output‑based policy improvements with policy evaluations according to a prescribed periodic pattern τℓ (ℓ = 0,1,…). The pattern determines which time steps are updated in each iteration. For a given τℓ, the algorithm proceeds as follows:

1. Policy Evaluation – For each stage t = τℓ, compute the observation‑action value
   (\bar Q^{\pi_\ell}t(o,a)=\sum_s Q^{\pi\ell}_t(s,a),\alpha_t(s|o))
   where (\alpha_t(s|o)=\Pr{S_t=s\mid O_t=o}) is obtained via Bayes’ rule, and also propagate the state distribution (\mu_t) forward using the dynamics.

2. Policy Improvement – Update the deterministic decision rule at the same stage by
   (\pi_{\ell+1}(t)(o)=\arg\max_a \bar Q^{\pi_\ell}_t(o,a)).
   All other stages retain their previous actions.

The key theoretical contributions are:

• Monotonicity and Convergence (Theorem 1). If the sequence τℓ is onto (every stage is eventually visited) and never repeats the same stage consecutively, the expected episodic return (L_{\pi_\ell}) never decreases and the algorithm terminates at a locally optimal memoryless policy—i.e., no single‑stage deterministic deviation can improve the return.

• Computational‑Efficiency Index. The authors define an index counting the number of policy‑evaluation operations required per period. By analyzing how forward‑propagated state distributions (\mu_t) and backward‑propagated Q‑values interact, they show that after improving a stage τ₀, only the forward‑propagated (\mu) between τ₀ + 1 and the next updated stage τ₁ need recomputation, while the Q‑values for earlier stages can be reused via backward recursion. This insight dramatically reduces redundant work.

• Optimal Periodic Patterns. Comparing forward sweeps (0→T‑1) and backward sweeps (T‑1→0) reveals that the minimal‑period pattern (period M = 1) that alternates updates across stages yields the smallest index. In this pattern each iteration updates exactly one stage, requiring only a narrow band of (\mu) or Q recomputations, leading to an overall linear‑in‑T cost instead of quadratic.

• Model‑Free Extension. The same periodic framework is adapted to reinforcement‑learning settings where the transition, observation, and reward models are unknown. The authors replace the exact (\alpha_t) and Q‑values with sample‑based estimators (e.g., TD‑learning for (\bar Q) and empirical Bayes updates for (\mu)). Policy improvement still follows the arg‑max rule, possibly with ε‑greedy or softmax exploration. Because only one stage is touched per iteration, the data efficiency is markedly higher than that of full‑trajectory policy‑gradient methods.

Empirical Evaluation. The authors test both the model‑based and model‑free versions on classic POMDP benchmarks (Tiger, RockSample, Maze) and a simulated robot navigation task. In the model‑based regime, their method outperforms the recent MILP‑based approach of Cohen & Parlementier (2023) by a factor of 2–5 in wall‑clock time while achieving 90‑95 % of the optimal return. In the model‑free regime, it converges 3–7 times faster than REINFORCE, PPO, and other policy‑gradient baselines, and shows greater robustness to observation noise. Ablation studies confirm that increasing the period length M yields diminishing returns in solution quality but a steep rise in computation, validating the theoretical optimality of the minimal‑period pattern.

Significance and Outlook. The paper’s central insight—that non‑Markovian output processes can still be handled by a carefully orchestrated, periodic update schedule—opens a new avenue for scalable POMDP planning with deterministic, interpretable policies. Deterministic memoryless policies are especially attractive for safety‑critical domains (autonomous driving, robotics) where predictability is paramount. Future work suggested includes extending the framework to stochastic memoryless policies, integrating with hierarchical options, and deploying on real‑time embedded platforms with strict latency constraints.

In sum, the authors deliver a theoretically grounded, computationally efficient, and empirically validated policy‑iteration algorithm that brings memoryless POMDP control from a theoretical curiosity to a practical tool for both model‑based planning and model‑free reinforcement learning.