Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting

Probabilistic forecasting is increasingly critical across high-stakes domains, from finance and epidemiology to climate science. However, current evaluation frameworks lack a consensus metric and suffer from two critical flaws: they often assume independence across time steps or variables, and they demonstrably lack sensitivity to tail events, the very occurrences that are most pivotal in real-world decision-making. To address these limitations, we propose two kernel-based metrics: the signature maximum mean discrepancy (Sig-MMD) and our novel censored Sig-MMD (CSig-MMD). By leveraging the signature kernel, these metrics capture complex inter-variate and inter-temporal dependencies and remain robust to missing data. Furthermore, CSig-MMD introduces a censoring scheme that prioritizes a forecaster’s capability to predict tail events while strictly maintaining properness, a vital property for a good scoring rule. These metrics enable a more reliable evaluation of direct multi-step forecasting, facilitating the development of more robust probabilistic algorithms.

💡 Research Summary

**
Probabilistic forecasting has become indispensable in high‑stakes domains such as finance, epidemiology, climate science, and energy management. Yet the community still lacks a consensus evaluation metric, and the most widely used scores—Quantile Loss, Continuous Ranked Probability Score (CRPS), Energy Score (ES), and Variogram Score (VS)—suffer from two fundamental shortcomings. First, they implicitly assume independence across time steps and across variables, thereby ignoring the rich temporal and cross‑variable dependencies that modern multivariate forecasting models aim to capture. Second, they are largely insensitive to tail events, i.e., low‑probability but high‑impact outcomes that are precisely what decision‑makers care about.

To address these gaps, Redhead et al. introduce two kernel‑based scoring rules that operate directly on samples generated by a forecaster: Signature Maximum Mean Discrepancy (Sig‑MMD) and its censored variant (CSig‑MMD). Both metrics rely on the signature transform, a path‑wise feature map that encodes a time series as an infinite collection of iterated integrals. The signature is invariant to re‑parameterisation, uniquely characterises the ordering and geometry of a trajectory, and can be embedded into a Reproducing Kernel Hilbert Space (RKHS) via the signature kernel (k_{\text{sig}}(x,y)=\langle S(x),S(y)\rangle). When combined with a characteristic kernel (the authors use an RBF kernel as a static base), the resulting kernel is characteristic, guaranteeing that the associated Maximum Mean Discrepancy (MMD) is a strictly proper scoring rule. Importantly, MMD does not require an explicit density; it can be estimated from finite samples, making it suitable for modern generative forecasters that output Monte‑Carlo trajectories rather than analytical densities.

Signature‑MMD (Sig‑MMD).
The authors first augment each multivariate series with a time coordinate, a base‑point, and an end‑point to ensure sensitivity to temporal shifts and translations. The augmented paths are fed into the signature kernel, producing a kernel matrix whose entries are inner products of signature features. The MMD squared between the ground‑truth distribution (P) and the forecast distribution (Q) is then computed as (| \mu_P - \mu_Q |^2_{\mathcal H_{\text{sig}}}), where (\mu_P) and (\mu_Q) are the mean embeddings in the RKHS. Because the signature kernel can be evaluated via a Goursat PDE (the “kernel trick”), the computational cost scales linearly with the total length of the two paths and the ambient dimension, avoiding the exponential blow‑up that a naïve truncated signature would incur.

Sig‑MMD captures the full joint distribution over all forecast horizons and variables, thereby addressing the independence assumption that plagues CRPS, ES, and Quantile Loss. In synthetic experiments, the authors demonstrate that models which correctly learn cross‑step covariances receive significantly lower Sig‑MMD scores even when their marginal distributions are similar to those of poorer models.

Censored‑Signature‑MMD (CSig‑MMD).
While Sig‑MMD is globally sensitive, it does not prioritize rare extreme outcomes. To create a tail‑focused yet proper scoring rule, the authors adopt the notion of distribution censoring. They first compute a Mahalanobis distance for each signature‑truncated path with respect to the empirical mean (\mu) and covariance (\Sigma) of the ground‑truth signatures. A soft logistic weight \