Evaluation of clinical utility in emulated clinical trials

Dynamic treatment regimes have been proposed to personalize treatment decisions by utilizing historical patient data, but they may not always improve on the current standard of care. It is thus meaningful to integrate the standard of care into the evaluation of treatment strategies, and previous works have suggested doing so through the concept of clinical utility. Here we will focus on the comparative component of clinical utility as the average outcome had the full population received treatment based on the proposed dynamic treatment regime in comparison to the full population receiving the standard" treatment assignment mechanism, such as a physician's choice. Clinical trials to evaluate clinical utility are rarely conducted, and thus, previous works have proposed an emulated clinical trial framework using observational data. However, only one simple estimator was previously suggested, and the practical details of how one would conduct this emulated trial were not detailed. Here, we illuminate these details and propose several estimators of clinical utility based on estimators proposed in the dynamic treatment regime literature. We illustrate the considerations and the estimators in a real data example investigating treatment rules for rheumatoid arthritis, where we highlight that in addition to the standard of care, the current medical guidelines should also be compared to any estimated optimal’’ decision rule.

💡 Research Summary

This paper addresses a critical gap in personalized medicine research: evaluating whether a proposed dynamic treatment regime (DTR) offers genuine clinical value over the current standard of care. While DTRs aim to personalize treatment using patient covariates, their superiority to existing practice is often assumed rather than rigorously tested. The authors formalize this comparison through the concept of “clinical utility,” defined as the difference in expected population outcomes under the proposed DTR versus under the standard treatment assignment mechanism (e.g., physician’s choice or existing guidelines).

Since dedicated randomized trials for this purpose are rare, the paper advocates for an “emulated clinical trial” framework using observational data. It moves beyond previous theoretical suggestions by providing practical details for implementation and proposing a suite of concrete estimators for clinical utility. These estimators are adapted from the causal inference literature on DTRs and include two inverse probability weighting (IPW) estimators—one based on a multinomial model of the observed treatment (ipw_nb), and another based on a binary model of adherence to the DTR rule (ipw_b)—and two g-computation estimators (gc_nb, gc_b). The choice among them involves trade-offs: multinomial IPW leverages subject-matter knowledge about treatment assignment but can be unstable, while binary IPW uses standard models but estimates a less intuitive propensity. G-computation requires correct specification of the outcome model but avoids weight instability.

A comprehensive simulation study investigates the finite-sample properties of these estimators under varying sample sizes (n=200, 500, 2000) and scenarios where the DTR is optimal, equivalent, or sub-optimal compared to standard care. Key findings show that with large samples and correctly specified models, all estimators perform well. However, with smaller samples, the IPW estimators exhibit higher variance and under-coverage of confidence intervals, whereas the g-computation estimators remain more stable. Model misspecification degrades performance for all methods, underscoring the importance of careful model diagnostics.

The methodology is illustrated using a real-world data example concerning treatment decisions for rheumatoid arthritis (RA). This case study highlights crucial practical considerations when emulating a trial, such as defining a time-fixed eligibility point, adjusting for potential confounders, and, most importantly, carefully defining the “standard of care” for comparison. The authors emphasize that this standard can be dynamic (changing over time) and may differ from published clinical guidelines. The analysis demonstrates how the proposed estimators can be used to compare an estimated optimal DTR against both the observed physician decisions and the guideline-recommended treatment rule.

In conclusion, this paper provides a robust methodological framework and practical tools for evaluating the clinical utility of personalized treatment rules. By shifting the focus from pure optimization to comparative effectiveness against real-world practice, it enhances the translational relevance of DTR research and offers a more realistic assessment of their potential impact on patient care.