A game-theoretic version of Oakes example for randomized forecasting

Using the game-theoretic framework for probability, Vovk and Shafer. have shown that it is always possible, using randomization, to make sequential probability forecasts that pass any countable set of well-behaved statistical tests. This result generalizes work by other authors, who consider only tests of calbration. We complement this result with a lower bound. We show that Vovk and Shafer’s result is valid only when the forecasts are computed with unrestrictedly increasing degree of accuracy. When some level of discreteness is fixed, we present a game-theoretic generalization of Oakes’ example for randomized forecasting that is a test failing any given method of deferministic forecasting; originally, this example was presented for deterministic calibration.

💡 Research Summary

The paper investigates the limits of randomized forecasting within the game‑theoretic probability framework introduced by Vovk and Shafer. Their seminal result states that, by using randomization, a forecaster can produce sequential probability forecasts that pass any countable collection of well‑behaved statistical tests. This theorem, however, implicitly assumes that the forecaster can output probabilities with arbitrarily fine precision—essentially an unrestrictedly increasing degree of accuracy. The authors of the present work ask whether this assumption is essential, and they answer in the affirmative by providing a matching lower bound.

The key contribution is a game‑theoretic generalization of Oakes’ classic example, originally devised to show that deterministic calibration is impossible for certain adversarial sequences. In the present setting, the authors consider a three‑player game involving Nature, a Forecaster, and a Tester. Nature generates a binary outcome sequence, the Forecaster at each round announces a probability chosen from a fixed discrete grid ({0, \Delta, 2\Delta, \dots, 1}) (where (\Delta>0) is a predetermined step size), and the Tester simultaneously runs a countable family of “well‑behaved” statistical tests (including calibration, strong calibration, and various martingale‑type tests).

The authors construct an adversarial strategy for Nature that, for any forecaster constrained to the grid, forces the cumulative discrepancy (\sum_{t=1}^{n}(X_t-p_t)) to grow at least linearly in (n) with a slope proportional to (\Delta). Concretely, at each round Nature observes the announced probability (p_t) and then selects the outcome (X_t) to be 1 if (p_t<\frac12) and 0 otherwise (or any analogous rule that guarantees a minimum mismatch). Because the forecaster’s probability cannot be refined beyond the grid, the absolute error (|X_t-p_t|) is bounded below by (\Delta) on every round, leading to a cumulative error of at least (\Delta n).

Given this construction, any test in the countable family that satisfies the usual “well‑behaved” condition—such as (\limsup_{n\to\infty}\frac{1}{n}\bigl|\sum_{t=1}^{n}(X_t-p_t)\bigr|=0)—will be violated on the adversarial sequence. Hence, no deterministic or randomized forecasting method that is limited to a fixed discretization can simultaneously pass all such tests. The result demonstrates that the Vovk‑Shafer upper bound is tight only when the forecaster’s precision can increase without bound; if a level of discreteness is fixed, the upper bound collapses.

Beyond the technical theorem, the paper discusses several practical implications. Real‑world forecasting systems often operate with limited precision due to hardware constraints (e.g., 8‑bit floating‑point representations) or policy requirements (e.g., probabilities reported to two decimal places). In such environments, the adversarial construction shows that an opponent who can adaptively choose outcomes can systematically expose systematic mis‑calibration or other statistical deficiencies, regardless of how the forecaster randomizes its predictions. Consequently, practitioners should not rely solely on randomization to guarantee statistical validity. Instead, they may need to increase the granularity of probability outputs, employ adaptive grid refinement, or supplement randomization with additional defensive mechanisms such as post‑hoc calibration adjustments or robust test designs that account for discretization effects.

The paper also situates its contribution within the broader literature. Earlier works focused mainly on calibration tests; Vovk and Shafer extended the analysis to arbitrary countable families of tests, establishing a powerful positive result. The present work complements that line by providing a matching negative result, thereby delineating the exact conditions under which the positive theorem holds. By translating Oakes’ deterministic counterexample into the game‑theoretic, randomized setting, the authors reveal that randomization does not automatically neutralize adversarial sequences when the forecaster’s output space is coarse.

In summary, the authors prove that a fixed level of discreteness in probability forecasts imposes a fundamental limitation: any such forecasting scheme can be defeated by an appropriately designed adversarial sequence, causing failure of all countable well‑behaved statistical tests. The result underscores the necessity of arbitrarily fine precision for the universal validity of randomized forecasting in the game‑theoretic framework, and it offers concrete guidance for the design of robust forecasting systems in practice.