POLAR: A Pessimistic Model-based Policy Learning Algorithm for Dynamic Treatment Regimes

Dynamic treatment regimes (DTRs) provide a principled framework for optimizing sequential decision-making in domains where decisions must adapt over time in response to individual trajectories, such as healthcare, education, and digital interventions. However, existing statistical methods often rely on strong positivity assumptions and lack robustness under partial data coverage, while offline reinforcement learning approaches typically focus on average training performance, lack statistical guarantees, and require solving complex optimization problems. To address these challenges, we propose POLAR, a novel pessimistic model-based policy learning algorithm for offline DTR optimization. POLAR estimates the transition dynamics from offline data and quantifies uncertainty for each history-action pair. A pessimistic penalty is then incorporated into the reward function to discourage actions with high uncertainty. Unlike many existing methods that focus on average training performance or provide guarantees only for an oracle policy, POLAR directly targets the suboptimality of the final learned policy and offers theoretical guarantees, without relying on computationally intensive minimax or constrained optimization procedures. To the best of our knowledge, POLAR is the first model-based DTR method to provide both statistical and computational guarantees, including finite-sample bounds on policy suboptimality. Empirical results on both synthetic data and the MIMIC-III dataset demonstrate that POLAR outperforms state-of-the-art methods and yields near-optimal, history-aware treatment strategies.

💡 Research Summary

The paper introduces POLAR, a pessimistic model‑based offline reinforcement‑learning algorithm designed for dynamic treatment regimes (DTRs). Traditional statistical DTR methods rely on a strong positivity assumption—every history‑action pair must be observed with non‑zero probability—which is often violated in real‑world medical data due to clinical guidelines, physician preferences, or resource constraints. Offline RL approaches mitigate distribution shift by being conservative, but many are model‑free, lack finite‑sample statistical guarantees, and require solving complex minimax or constrained optimization problems.

POLAR addresses these gaps by first estimating the transition dynamics from a static offline dataset using maximum‑likelihood or other suitable estimators. For each history‑action pair ((h_k, a_k)) it constructs an uncertainty quantifier (\Gamma_k(h_k, a_k)) that upper‑bounds the L1 distance between the estimated transition kernel (\hat P_k) and the true kernel (P_k^*) with high probability (Assumption 1). Concrete forms are provided for linear transition models (using ridge‑regularized feature matrices) and Gaussian‑process models (using covering numbers and noise variance).

The algorithm then builds a pessimistic reward function \