Safe, Always-Valid Alpha-Investing Rules For Doubly Sequential Online Inference

Dynamic decision-making in rapidly evolving research domains, including marketing, finance, and pharmaceutical development, presents a significant challenge. Researchers frequently confront the need for real-time action within a doubly sequential framework characterized by the continuous influx of high-volume data streams and the intermittent arrival of novel tasks. This calls for the development and implementation of new online inference protocols capable of handling both the continuous processing of incoming information and the efficient allocation of resources to address emerging priorities. We introduce a novel class of Safe and Always-Valid Alpha-investing (SAVA) rules that leverages powerful tools including always valid p-values, e-processes, and online false discovery rate methods. The SAVA algorithm effectively integrates information across all tasks, mitigates the alpha-death problem, and controls the false selection rate (FSR) at all decision points. We validate the efficacy of the SAVA framework through rigorous theoretical analysis and extensive numerical experiments. Our results demonstrate that SAVA not only offers effective control of the FSR but also significantly improves statistical power compared to traditional online testing approaches.

💡 Research Summary

The paper addresses a novel “doubly sequential” setting in which multiple experiments (tasks) arrive over time and each task generates its own data stream that is also observed sequentially. This asynchronous, overlapping structure is common in modern high‑throughput domains such as online advertising A/B testing, financial customer‑screening pipelines, and high‑throughput drug discovery. Existing methods either focus on classical sequential testing (which assumes a single data stream) or on online false discovery rate (FDR) control (which assumes a synchronous stream of hypotheses). Both fail to handle the simultaneous temporal and task‑level sequencing, especially when new tasks can start before earlier ones finish and when the timing of arrivals and departures is unpredictable.

To fill this gap, the authors propose a class of Safe‑Always‑Valid Alpha‑Investing (SAVA) rules. The key ingredients are:

1. Always‑valid p‑values and e‑processes – statistical evidence that remains valid under arbitrary optional stopping, allowing continuous updating of test statistics as data arrive.
2. Expanded action space – each task at any decision time can choose among four actions: declare arm A superior, declare arm B superior, continue data collection (abstention C), or drop the task (abstention D). The abstention options prevent premature decisions and give the algorithm flexibility to allocate resources adaptively.
3. Alpha‑investing with wealth recycling – successful discoveries generate “alpha‑wealth” that is immediately redistributed to other active tasks, thereby avoiding the classic “alpha‑death” problem where the testing budget is exhausted.
4. Leave‑sequence‑out analysis – a novel extension of the leave‑one‑out technique that accounts for the dependence between tasks’ statistics caused by shared wealth dynamics. This enables finite‑sample proofs that the false selection rate (FSR) is controlled at any time point, even with directional errors.

The theoretical contributions are substantial. The authors prove that under the SAVA rules, the FSR never exceeds a pre‑specified level α, regardless of the stochastic arrival pattern of tasks, the lengths of data streams, or the adaptive stopping times used. The proof relies on martingale properties of always‑valid p‑values/e‑processes and on showing that the wealth process is a non‑negative super‑martingale, guaranteeing that the testing budget never becomes negative.

Empirically, the paper evaluates SAVA on synthetic simulations and on real‑world datasets from online advertising and high‑throughput screening. Compared with state‑of‑the‑art online FDR procedures such as LORD, SAFFRON, and the original alpha‑investing methods, SAVA consistently maintains tighter FSR control while achieving 10–30 % higher power. In scenarios with high task density, where traditional methods suffer from rapid alpha‑wealth depletion, SAVA continues to make discoveries throughout the experiment horizon, demonstrating its robustness to “alpha‑death.”

Limitations are acknowledged: (i) constructing always‑valid p‑values may require specific modeling choices (e.g., martingale constructions or Bayesian priors); (ii) the richer action space and wealth‑recycling mechanism increase computational overhead relative to simple online FDR rules; (iii) the current formulation focuses on binary comparisons with directional tests, leaving extensions to multi‑arm or composite hypotheses for future work.

In summary, the Safe‑Always‑Valid Alpha‑Investing framework offers a principled, theoretically sound, and practically powerful solution for real‑time inference in doubly sequential experiments. By integrating always‑valid evidence, adaptive abstention, and dynamic wealth management, SAVA enables continuous, error‑controlled decision making across overlapping tasks, addressing a critical gap in modern large‑scale experimental platforms.