Decision-Focused Sequential Experimental Design: A Directional Uncertainty-Guided Approach

We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.

💡 Research Summary

The paper tackles the sequential experimental design problem in the predict‑then‑optimize framework, where a predictive model supplies the cost coefficients of a downstream linear optimization problem. Traditional sequential design focuses on maximizing information gain about the unknown regression function, typically measured by reductions in prediction error (e.g., mean‑squared error). However, when the ultimate performance metric is the decision loss induced by the downstream optimization (the SPO loss), reducing prediction error does not necessarily translate into better decisions. This mismatch makes conventional, decision‑blind designs inefficient.

To bridge this gap, the authors introduce a directional‑uncertainty metric. The key observation (Fact 1) is that in linear optimization the scale of the cost vector does not affect the optimal decision; only its direction matters. Accordingly, each predicted cost vector is normalized onto the unit ℓ₂‑ball, and the maximum pairwise distance between these normalized predictions over the current hypothesis class Hₜ is taken as the uncertainty of a candidate design x. This “directional uncertainty” discards irrelevant magnitude information while preserving the components that influence the downstream decision. Importantly, computing this metric requires no calls to the downstream optimization oracle, unlike a direct SPO‑based uncertainty measure, which would be computationally prohibitive.

The authors provide a rigorous theoretical analysis. Assuming a finite hypothesis class, they prove strong consistency of the sequential policy that selects the design with the largest directional uncertainty at each round. They also establish non‑asymptotic excess‑risk bounds and show that the stopping time τ*—the first iteration at which the expected SPO risk falls below a prescribed ε—is, in expectation, earlier for the directional‑uncertainty policy than for any decision‑blind baseline that relies solely on prediction‑error uncertainty. A crucial “suboptimality gap” assumption (there exists a positive gap in expected decision loss between optimal and suboptimal predictors) enables a formal comparison of stopping times; the authors demonstrate that this assumption holds for a broad family of distributions (e.g., Gaussian, uniform, Laplace).

Empirically, the method is evaluated on two real‑world problems. The first involves cardiovascular disease diagnosis where patient features are used to predict treatment cost vectors; the downstream problem is a linear resource allocation model. The second is a contextual job‑assignment task for large language models (LLMs), where prompts must be allocated to models of varying capability and cost under capacity constraints. In both settings, the directional‑uncertainty design achieves the same SPO risk as traditional methods while requiring roughly 30–50 % fewer labeled samples. In the LLM experiment, this translates into a substantial reduction in costly human evaluations of model performance on prompts.

Overall, the contribution is threefold: (1) a novel, computationally cheap uncertainty metric aligned with decision loss for linear predict‑then‑optimize problems; (2) strong theoretical guarantees showing earlier stopping and excess‑risk control compared with decision‑blind designs; (3) convincing empirical evidence that the approach yields significant data‑efficiency gains in high‑stakes applications where labeling is expensive. The work opens a promising direction for decision‑focused data acquisition in settings where downstream optimization is the true objective.