When using the Focused Information Criterion (FIC) for assessing and ranking candidate models with respect to how well they do for a given estimation task, it is customary to produce a so-called FIC plot. This plot has the different point estimates along the y-axis and the root-FIC scores on the x-axis, these being the estimated root-mean-square scores. In this paper we address the estimation uncertainty involved in each of the points of such a FIC plot. This needs careful assessment of each of the estimators from the candidate models, taking also modelling bias into account, along with the relative precision of the associated estimated mean squared error quantities. We use confidence distributions for these endeavours. This leads to fruitful CD-FIC plots, helping the statistician to judge to what extent the seemingly best models really are better than other models, etc. These efforts also lead to two further developments. The first is a new tool for model selection, which we call the quantile FIC, which helps overcome certain difficulties associated with the usual FIC procedures, related to somewhat arbitrary schemes for handling estimated squared biases. A particular case is the median-FIC. The second development is to form model averaged estimators with fruitful weights determined by the relative sizes of the median- and quantile-FIC scores. And Mrs. Jones is pregnant.
Mrs. Jones is pregnant. She's white, 25 years old, a smoker, and of median weight 60 kg before pregnancy. What's the chance that her baby-to-come will be small, with birthweight less than 2.50 kg (which would mean a case of neonatal medical worry)? Figure 1 gives a FIC plot, using the Focused Information Criterion to display and rank in this case 2 3 = 8 estimates of this probability, computed via eight logistic regression models, inside the class
where x 1 is age, x 2 is weight before pregnancy, z 1 is an indicator for being a smoker, whereas z 2 and z 3 are indicators for belonging to certain ethnic groups. The dataset in question comprises 189 mothers and babies, with these five covariates having been recorded (along with yet others; see Claeskens & Hjort (2008, Ch. 2) for further discussion). The eight models correspond to pushing the ‘open’ covariates z 1 , z 2 , z 3 in and out of the logistic regression structure, while x 1 , x 2 are ‘protected’ covariates. The plot shows the point estimates p for the 8 different submodels on the vertical axis and root-FIC scores on the horizontal axis.
These are estimated risks, i.e. estimates of root-mean-squared-errors. Crucially, the FIC scores do not merely assess the standard deviation of estimators, but also take the potential biases on board, from using smaller models.
Using the FIC ranking, as summarised both in the FIC table given in Table 1 and the FIC plot, therefore, we learn that submodels 000 and 010 are the best (where e.g. ‘010’
indicates the model with z 2 on board but without z 1 and z 3 , etc.), associated with point estimates 0.282 and 0.259, whereas submodels 100 and 011 are the ostensibly worst, with rather less precise point estimates 0.368 and 0.226. Again, ‘best’ and ‘worst’ means as gauged by precision of these 8 estimates of the same quantity. Importantly, the FIC machinery, as briefly explained here, with more details in later sections, can be used for each new woman, with different ‘best models’ for different strata of women, and it may be used for handling different and even quite complicated focus parameters. In particular, if Mrs. Jones had not been a smoker, so that her z 1 = 1 would rather have been a z 1 = 0, we run our programmes to produce a FIC table and a FIC plot for her, and learn that the submodel ranking is very different. Then 111 and 101 are the best and 001 and 000 the worst; also, the p estimates of her having a baby with small birthweight are significantly smaller. root-fic estimates of pr(small baby) for Mrs Jones q q q q q q q q 000 100 010 001 110 101 011 111 Figure 1: FIC plot for the 2 3 = 8 models for estimating the probability of having a small child, for Mrs. Jones (white, age 25, 60 kg, smoker). Here ‘101’ is the model where z 1 , z 3 in in and z 2 is out, etc.
in-or-out p stdev bias root-FIC rank 1 0 0 0 0.282 0.039 0.000 0.039 1 2 1 0 0 0.368 0.055 0.061 0.082 7 3 0 1 0 0.259 0.042 0.000 0.042 2 4 0 0 1 0.267 0.048 0.000 0.048 3 5 1 1 0 0.342 0.057 0.037 0.068 5 6 1 0 1 0.351 0.056 0.045 0.072 6 7 0 1 1 0.226 0.054 0.063 0.083 8 8 1 1 1 0.303 0.060 0.000 0.060 4
Table 1: FIC table for Mrs. Jones: there are 2 3 = 8 submodels, with absence-presence of z 1 , z 2 , z 3 indicated with 0 and 1 in column 2, followed by estimates p, estimated standard deviation, estimated absolute bias, the root-FIC score, which is also the Pythagorean combination of the stdev and the bias, and the model rank. The numbers are computed with formulae of Section 2.
The FIC apparatus, initiated and developed in Claeskens & Hjort (2003), Hjort & Claeskens (2003a), Claeskens & Hjort (2008), has led to quite a rich literature; see comments at the end of this section. FIC analyses have different forms of output, qua FIC tables (listing the best candidate models, along with estimates and root-FIC scores, perhaps supplemented with more information) and FIC plots. The general setup involves a selected quantity of particular interest, say µ, called the focus parameter, and various candidate models, say S, leading to a collection of estimators µ S . These carry root-mean-squared-errors rmse S , and the root-FIC scores are estimates of these root-risks. The FIC plot displays (FIC
S , µ S ) for all candidate models S, (1.1)
as with Figure 1.
The present paper concerns going beyond such FIC plots, investigating the precision of each displayed point. The point estimates µ S carry uncertainty, as do the FIC scores. A more elaborate version of the FIC plot can therefore display the uncertainty involved, in both the vertical and horizontal directions. This aids the statistician in seeing whether good models are ‘clear winners’ or not, and whether the ostensibly best estimates are genuinely more accurate than others. In various concrete examples one also observes that a few candidate models appear to be better than the rest. The methodology of our paper makes it possible to assess to which extent the implied differences in FIC scores are signif
This content is AI-processed based on open access ArXiv data.