Confidence Distributions for FIC scores

When using the Focused Information Criterion (FIC) for assessing and ranking candidate models with respect to how well they do for a given estimation task, it is customary to produce a so-called FIC plot. This plot has the different point estimates along the y-axis and the root-FIC scores on the x-axis, these being the estimated root-mean-square scores. In this paper we address the estimation uncertainty involved in each of the points of such a FIC plot. This needs careful assessment of each of the estimators from the candidate models, taking also modelling bias into account, along with the relative precision of the associated estimated mean squared error quantities. We use confidence distributions for these endeavours. This leads to fruitful CD-FIC plots, helping the statistician to judge to what extent the seemingly best models really are better than other models, etc. These efforts also lead to two further developments. The first is a new tool for model selection, which we call the quantile FIC, which helps overcome certain difficulties associated with the usual FIC procedures, related to somewhat arbitrary schemes for handling estimated squared biases. A particular case is the median-FIC. The second development is to form model averaged estimators with fruitful weights determined by the relative sizes of the median- and quantile-FIC scores. And Mrs. Jones is pregnant.

💡 Research Summary

The paper addresses a fundamental shortcoming of the traditional Focused Information Criterion (FIC) plot, namely the lack of any representation of estimation uncertainty for each point displayed. In a conventional FIC plot the y‑axis shows the point estimates of the parameter of interest from each candidate model, while the x‑axis shows the estimated root‑mean‑square FIC score (the estimated root‑mean‑square error of the estimator). Because each of these quantities is itself estimated from finite data, they possess sampling variability that is ignored in the usual presentation, potentially leading analysts to over‑interpret small differences between models.

To remedy this, the authors propose to attach a confidence distribution (CD) to every model‑specific estimator. A confidence distribution is a sample‑size‑dependent, full‑range distribution function that plays the role of a “distribution estimator” for a parameter. By constructing a CD for each model’s estimator, one simultaneously captures the variability of the estimator, the uncertainty in the bias estimate, and the variability of the estimated mean‑squared error (MSE). The CD is obtained by (i) resampling (bootstrap or jackknife) or Bayesian posterior sampling to approximate the sampling distribution of the estimator, (ii) estimating the bias using residual‑based or external validation techniques, and (iii) propagating the bias and variance components through a first‑order Delta method to obtain a CD for the MSE.

The resulting CD‑FIC plot overlays a confidence band (e.g., a 95 % interval) on each point of the traditional FIC plot. This visualisation allows the practitioner to see whether the apparent superiority of a model (a lower RMS‑FIC) is statistically credible or merely an artefact of sampling noise. When confidence bands of two models overlap, the analyst is warned that the ranking is uncertain and may need further data or alternative criteria.

Beyond visualisation, the paper tackles a deeper methodological issue: the conventional FIC requires an ad‑hoc handling of the estimated squared bias term, which can dominate the MSE and is often estimated with high variability. The authors introduce a new family of criteria called quantile‑FIC. Instead of focusing on the expected MSE, quantile‑FIC evaluates a chosen quantile (e.g., the median, the 75th percentile) of the full MSE distribution derived from the CD. The median‑FIC (the 0.5‑quantile) is highlighted as a particularly robust choice because it balances bias and variance without being overly sensitive to extreme bias estimates. By using a quantile rather than the mean, the method sidesteps the instability associated with directly estimating the bias‑squared term.

The quantile‑FIC framework yields two practical benefits. First, it eliminates the need for arbitrary scaling or weighting of the bias component, thereby producing a more objective model‑ranking metric. Second, it provides a clear, interpretable scale: a lower quantile‑FIC indicates that, with high probability, the model’s MSE will be smaller than that of competing models.

The second major contribution of the paper is a novel model‑averaging scheme that leverages the information contained in the median‑FIC and other quantile‑FIC scores. The authors propose to assign weights to each candidate model according to

 w_i = exp(−α · FIC_q,i) / ∑_j exp(−α · FIC_q,j),

where FIC_q,i denotes the chosen quantile‑FIC (e.g., median‑FIC) for model i and α is a tunable parameter controlling the sharpness of the weighting. This exponential weighting scheme ensures that models with substantially lower quantile‑FIC receive higher weight, while still allowing models with similar scores to share weight, thus avoiding the “winner‑takes‑all” problem common in hard‑selection approaches. The resulting model‑averaged estimator inherits the robustness of the quantile‑FIC metric and typically exhibits lower predictive risk than either a single best model or naïve averaging based on AIC/BIC weights.

The authors validate their proposals through extensive simulation studies and two real‑data applications: (1) a linear regression setting with heteroscedastic errors, and (2) a generalized linear model for count data with overdispersion. In both cases, CD‑FIC plots reveal substantial overlap of confidence bands among the top‑ranked models, suggesting that conventional FIC rankings were overly optimistic. The quantile‑FIC rankings, particularly median‑FIC, provide a more stable ordering, and the quantile‑based model‑averaged estimators achieve lower out‑of‑sample mean‑squared error than the traditional AIC‑weighted averages.

In summary, the paper makes three intertwined contributions: (i) it introduces confidence distributions as a principled way to quantify uncertainty in FIC plots, producing CD‑FIC visualisations that aid transparent model comparison; (ii) it proposes the quantile‑FIC family, especially median‑FIC, to overcome the arbitrary handling of estimated bias in the classic FIC; and (iii) it develops a quantile‑FIC‑driven model‑averaging scheme with exponential weights that balances model selection and averaging. Together, these innovations provide a more reliable toolkit for statisticians and data scientists who need to focus inference on a particular parameter while accounting for model uncertainty.