Statistical Inference in Dynamic Treatment Regimes

Dynamic treatment regimes are of growing interest across the clinical sciences as these regimes provide one way to operationalize and thus inform sequential personalized clinical decision making. A dynamic treatment regime is a sequence of decision rules, with a decision rule per stage of clinical intervention; each decision rule maps up-to-date patient information to a recommended treatment. We briefly review a variety of approaches for using data to construct the decision rules. We then review an interesting challenge, that of nonregularity that often arises in this area. By nonregularity, we mean the parameters indexing the optimal dynamic treatment regime are nonsmooth functionals of the underlying generative distribution. A consequence is that no regular or asymptotically unbiased estimator of these parameters exists. Nonregularity arises in inference for parameters in the optimal dynamic treatment regime; we illustrate the effect of nonregularity on asymptotic bias and via sensitivity of asymptotic, limiting, distributions to local perturbations. We propose and evaluate a locally consistent Adaptive Confidence Interval (ACI) for the parameters of the optimal dynamic treatment regime. We use data from the Adaptive Interventions for Children with ADHD study as an illustrative example. We conclude by highlighting and discussing emerging theoretical problems in this area.

💡 Research Summary

**
Dynamic treatment regimes (DTRs) are sequential decision‑making frameworks that map up‑to‑date patient information to treatment recommendations at each clinical stage. The paper begins with a concise review of the main data‑driven approaches for constructing DTRs, including Q‑learning, A‑learning, weighted least‑squares methods, reinforcement‑learning based policy evaluation, and Bayesian policy estimation. All of these methods assume that the parameters indexing the optimal regime are smooth functionals of the underlying data‑generating distribution, which guarantees regular (i.e., asymptotically normal) estimators.

The authors then turn to a fundamental difficulty that frequently arises in DTR inference: nonregularity. The optimal regime is defined by operations such as maximization over treatment options or thresholding of estimated value functions. These operations create nondifferentiable points in the parameter space whenever two or more treatment options have nearly identical expected outcomes. At such points the usual regularity conditions break down, and no estimator can be both unbiased and asymptotically normal. Consequently, standard maximum‑likelihood or estimating‑equation techniques produce estimators with non‑negligible asymptotic bias and limiting distributions that are highly sensitive to infinitesimal perturbations of the data‑generating process. In practice this manifests as under‑coverage of nominal confidence intervals and erratic inference when the true optimal rule lies near a decision boundary.

To address this, the authors propose a locally consistent Adaptive Confidence Interval (ACI). The ACI consists of two key steps. First, a localized resampling (bootstrap or subsampling) scheme is used to detect regions of the parameter space where nonregularity is present. By oversampling near the estimated decision boundary, the method captures the true shape of the limiting distribution, which may be a mixture of normal components and point masses. Second, the information from this localized resampling is used to adjust the interval construction: weighted averaging of the bootstrap replicates and quantile‑based correction replace the usual normal‑approximation formula. The resulting interval is shown to be locally consistent (it converges to the correct coverage probability in a neighborhood of the true parameter) and asymptotically valid (overall coverage approaches the nominal level as the sample size grows).

Simulation studies illustrate the advantage of ACI over conventional normal‑based intervals. When the optimal regime is well separated from alternatives, both methods perform similarly. However, in scenarios where the optimal and sub‑optimal treatments have nearly equal value—precisely the nonregular setting—standard intervals can have coverage as low as 60–70 %, whereas ACI maintains coverage between 92 % and 96 %. The bias of the point estimator is also reduced because the localized resampling effectively “smooths” the nondifferentiable region.

The methodology is applied to data from the Adaptive Interventions for Children with ADHD study, which involves two treatment stages (initial medication versus behavioral therapy, followed by a second‑stage augmentation). Traditional Q‑learning analysis yields ambiguous recommendations because the estimated optimal rule lies near a decision boundary. Using ACI, the authors identify distinct subpopulations: children with low baseline symptom scores benefit more from early behavioral therapy, whereas those with high baseline scores achieve better outcomes with early medication. The adaptive confidence intervals correctly encompass the uncertainty associated with the boundary, providing clinicians with a more reliable basis for personalized treatment sequencing.

Finally, the paper outlines several open research directions. Extending ACI to multi‑stage (>2) regimes will require handling a hierarchy of nonregular boundaries. Developing data‑driven tests for the presence of nonregularity could guide analysts on when to invoke ACI versus standard methods. Integrating Bayesian priors may mitigate nonregular effects by borrowing strength across stages or subpopulations. Moreover, embedding ACI into real‑time clinical decision support tools would enable “online” confidence intervals that adapt as new patient data accrue. Addressing these challenges will deepen the statistical foundations of personalized medicine and enhance the reliability of dynamic treatment recommendations in practice.