$Q$- and $A$-Learning Methods for Estimating Optimal Dynamic Treatment Regimes

In clinical practice, physicians make a series of treatment decisions over the course of a patient’s disease based on his/her baseline and evolving characteristics. A dynamic treatment regime is a set of sequential decision rules that operationalizes this process. Each rule corresponds to a decision point and dictates the next treatment action based on the accrued information. Using existing data, a key goal is estimating the optimal regime, that, if followed by the patient population, would yield the most favorable outcome on average. Q- and A-learning are two main approaches for this purpose. We provide a detailed account of these methods, study their performance, and illustrate them using data from a depression study.

💡 Research Summary

The paper addresses the problem of estimating optimal dynamic treatment regimes (DTRs) from existing data, a task of growing importance as clinicians repeatedly adjust therapies based on evolving patient information. Two principal statistical learning frameworks are examined: Q‑learning, which originates from reinforcement learning and focuses on modeling the full conditional expectation of future outcomes (the Q‑function), and A‑learning, an alternative that directly models the advantage function—the difference between the Q‑function for a specific action and the average value across actions.

The authors first lay out the formal definition of a DTR as a sequence of decision rules (\pi_t) that map the patient’s history at stage (t) to a treatment choice. They then derive the backward‑induction algorithm for Q‑learning: starting from the final stage, a regression of the observed outcome (or a pseudo‑outcome that incorporates the estimated value of later stages) on covariates and treatment yields an estimate of (Q_T); this estimate is fed back to earlier stages to obtain (Q_{T-1},\dots,Q_1). The optimal rule at each stage is simply the action that maximizes the estimated Q‑value. While conceptually straightforward, Q‑learning requires a correctly specified model for the entire Q‑function, making it vulnerable to misspecification, especially in high‑dimensional settings.

A‑learning mitigates this vulnerability by decomposing the Q‑function into a baseline value (V_t(S_t)=E