Positive-shrinkage and Pretest Estimation in Multiple Regression: A Monte Carlo study with Applications

Consider a problem of predicting a response variable using a set of covariates in a linear regression model. If it is \emph{a priori} known or suspected that a subset of the covariates do not significantly contribute to the overall fit of the model, a restricted model that excludes these covariates, may be sufficient. If, on the other hand, the subset provides useful information, shrinkage method combines restricted and unrestricted estimators to obtain the parameter estimates. Such an estimator outperforms the classical maximum likelihood estimators. Any \emph{prior} information may be validated through preliminary test (or pretest), and depending on the validity, may be incorporated in the model as a parametric restriction. Thus, pretest estimator chooses between the restricted and unrestricted estimators depending on the outcome of the preliminary test. Examples using three real life data sets are provided to illustrate the application of shrinkage and pretest estimation. Performance of positive-shrinkage and pretest estimators are compared with unrestricted estimator under varying degree of uncertainty of the prior information. Monte Carlo study reconfirms the asymptotic properties of the estimators available in the literature.

💡 Research Summary

The paper addresses the problem of estimating regression coefficients when there is prior (or suspected) information that a subset of covariates does not meaningfully contribute to the response. In the linear model Y = Xβ + ε, the coefficient vector β is partitioned into β₁ (the parameters of primary interest) and β₂ (the “nuisance” parameters). The prior belief that β₂ ≈ 0 is expressed as a linear restriction Hβ = h. Two families of estimators are considered: (i) a preliminary‑test (pre‑test) estimator that first tests the restriction using the χ²‑type statistic ψₙ = (Hβ̂_UR – h)′(HCH′)⁻¹(Hβ̂_UR – h)/ŝ_e² and then selects either the unrestricted least‑squares estimator β̂_UR or the restricted estimator β̂_R depending on whether ψₙ exceeds a critical value c_{n,α}; and (ii) a shrinkage estimator of Stein type, β̂_S₁ = β̂_R₁ + (β̂_UR₁ – β̂_R₁)(1 – κ/ψₙ)/n, where κ = p₂ – 2 (p₂ ≥ 3). Because the factor (1 – κ/ψₙ)/n can become negative, the authors introduce a “positive‑shrinkage” estimator that retains only the positive part: β̂_S⁺₁ = β̂_R₁ + (β̂_UR₁ – β̂_R₁)·max{0, 1 – κ/ψₙ}/n.

Theoretical properties of these estimators are derived under mild regularity conditions: asymptotic bias, variance, and quadratic risk expressions are obtained, showing that the positive‑shrinkage estimator dominates the ordinary Stein‑type estimator in risk and is less sensitive to the correctness of the prior restriction than the pre‑test estimator.

Empirical performance is illustrated with three real data sets. The first example uses the prostate cancer data (log‑PSA as response, several clinical measurements as predictors). Models are built using all variables (full model) and three sub‑models selected by AIC, BIC, and a best‑subset (BSS) criterion. Five‑fold and ten‑fold cross‑validation are repeated 5,000 times to obtain average prediction errors (both raw CVE and bias‑corrected CVE). Results show that when the sub‑model is well‑specified (AIC or BIC), the restricted and pre‑test estimators achieve the lowest errors, but for the poorly specified BSS sub‑model the positive‑shrinkage estimator yields substantially smaller errors, demonstrating its robustness to model misspecification.

The second example (the “state” data set) similarly confirms that positive‑shrinkage maintains stable prediction performance even when the chosen sub‑model omits important variables.

A comprehensive Monte‑Carlo simulation explores a range of scenarios: varying sample sizes, different degrees of departure of β₂ from zero, and differing levels of confidence in the prior restriction. The simulation confirms that (a) the pre‑test estimator performs best only when the prior is correct; otherwise its risk can explode, and (b) the positive‑shrinkage estimator consistently achieves lower or comparable risk across all settings, confirming its theoretical advantage.

The paper concludes that while pre‑test (or restricted) estimators are attractive when prior information is reliable, the positive‑shrinkage estimator offers a safer alternative that blends information from both the unrestricted and restricted models without the abrupt switch inherent in pre‑testing. It is less sensitive to misspecification, provides uniformly lower quadratic risk in many realistic settings, and is straightforward to implement. Future work is suggested on extending the methodology to high‑dimensional settings (e.g., integrating with LASSO), handling non‑normal error distributions, and developing Bayesian analogues.