Evaluating Weather Forecasts from a Decision Maker's Perspective

Standard weather forecast evaluations focus on the forecaster’s perspective and on a statistical assessment comparing forecasts and observations. In practice, however, forecasts are used to make decisions, so it seems natural to take the decision-maker’s perspective and quantify the value of a forecast by its ability to improve decision-making. Decision calibration provides a novel framework for evaluating forecast performance at the decision level rather than the forecast level. We evaluate decision calibration to compare Machine Learning and classical numerical weather prediction models on various weather-dependent decision tasks. We find that model performance at the forecast level does not reliably translate to performance in downstream decision-making: some performance differences only become apparent at the decision level, and model rankings can change among different decision tasks. Our results confirm that typical forecast evaluations are insufficient for selecting the optimal forecast model for a specific decision task.

💡 Research Summary

The paper introduces a decision‑centric evaluation framework—decision calibration—to assess weather forecasts based on the value they provide to downstream decision makers, rather than on traditional statistical skill scores alone. While conventional metrics such as the Continuous Ranked Probability Score (CRPS), Probability Integral Transform (PIT) histograms, and Spread‑Skill Ratios (SSR) quantify how well a forecast’s probability distribution matches observations, they do not directly address whether the forecast leads to better real‑world actions. Decision calibration reframes the problem: a cost function c(a, y) is defined for each decision task, the forecast’s predictive CDF Fₓ is used to compute the Bayes optimal action δ_c(Fₓ), and two quantities are estimated—expected cost under the forecast (C_exp) and observed cost after the true outcome is revealed (C_obs). Their absolute difference, the cost gap C_gap, serves as the decision‑level calibration metric. By focusing on cost gaps, the framework automatically emphasizes rare but high‑impact events and isolates the parts of the forecast distribution that matter for a given decision.

The authors apply this framework to compare a state‑of‑the‑art numerical weather prediction (NWP) ensemble (the ECMWF Integrated Forecast System Ensemble, IFS‑ENS) with a modern machine‑learning weather generator (ArchesweatherGen, a diffusion‑based model). Both systems produce 50‑member probabilistic forecasts for 2‑meter temperature and 10‑meter wind speed at lead times from 0 to 15 days, at 1.5° resolution, over the year 2021. Ground truth is taken from the IFS high‑resolution analysis (HRES) for the NWP and from ERA5 for the ML model.

Three realistic decision tasks are constructed, each with a simple yet plausible cost structure:

1. Frost protection in agriculture – a binary decision (protect vs. do nothing) based on a temperature threshold θ and a protection‑cost ratio c. If temperature falls below θ, crops are lost; protection incurs a cost proportional to c. The cost function is asymmetric, reflecting the higher penalty for missed protection.

2. Heat‑wave protection for civil protection – analogous binary decision with a higher‑temperature threshold and the same asymmetric cost formulation, representing the trade‑off between unnecessary heat‑wave measures and avoided mortality.

3. Wind‑power dispatch – a multi‑action decision where a wind farm promises a fraction of its rated power (11 discrete actions). Under‑delivery incurs a linear penalty with slope u‑pen, while over‑delivery can be curtailed at negligible cost. A small standby cost (“turbine‑off”) is also modeled.

For each grid point, forecast, and day, the expected cost of each possible action is approximated by Monte‑Carlo integration over the ensemble members, the optimal action is selected, and the resulting cost gap is computed by comparing to the observed outcome. The authors aggregate cost gaps over space and time, but also examine individual instances to avoid masking poor performance.

Key findings:

• Traditional skill scores (CRPS, PIT, SSR) show that IFS‑ENS and Arches are broadly comparable for temperature forecasts; Arches is slightly worse at 7–10 day lead times, and its ensembles tend to be under‑dispersed (narrower than observations) while IFS‑ENS can be over‑dispersed at longer leads.

• Decision calibration reveals nuanced differences. In the frost‑protection task, Arches outperforms IFS‑ENS for low temperature thresholds (i.e., when protection is needed in the far left tail of the distribution), but IFS‑ENS is superior for higher thresholds. This indicates that Arches provides more reliable probability estimates for extreme cold events, whereas IFS‑ENS better captures moderate temperature variability.

• In the heat‑wave task, performance varies with the chosen cost ratio c and threshold θ; neither model dominates across all configurations, highlighting that the relative importance of false alarms versus missed events can flip the ranking.

• For wind‑power dispatch, the penalty slope u‑pen strongly influences which model yields a smaller cost gap. With a steep penalty (high cost of under‑delivery), IFS‑ENS’s more conservative forecasts reduce the gap, whereas with a shallow penalty Arches’s higher expected generation leads to lower costs. Thus, the same forecast can be advantageous or detrimental depending on the economic context.

Overall, the study demonstrates that forecast‑level superiority does not guarantee decision‑level superiority. A model that scores better on CRPS may produce larger cost gaps for a particular application, and vice‑versa. Consequently, selecting the “best” weather forecast model for an operational setting must be based on decision‑calibrated evaluation tailored to the specific cost structure and action space of the end‑user.

Methodologically, the paper contributes by (i) formalizing decision calibration for continuous weather variables, (ii) proposing the cost‑gap metric as a practical, interpretable measure of decision‑level calibration, and (iii) showing the importance of evaluating forecasts at the individual instance level rather than only via aggregated averages.

The authors suggest future work on more complex, multi‑variable cost functions, incorporation of dynamic feedback where decisions affect subsequent forecasts, and extension to other sectors (e.g., aviation, water resources). Their findings advocate for a shift in the weather‑forecasting community toward decision‑oriented validation, ensuring that advances in model accuracy translate into tangible societal benefits.