Penalized Orthogonal-Components Regression for Large p Small n Data

We propose a penalized orthogonal-components regression (POCRE) for large p small n data. Orthogonal components are sequentially constructed to maximize, upon standardization, their correlation to the response residuals. A new penalization framework, implemented via empirical Bayes thresholding, is presented to effectively identify sparse predictors of each component. POCRE is computationally efficient owing to its sequential construction of leading sparse principal components. In addition, such construction offers other properties such as grouping highly correlated predictors and allowing for collinear or nearly collinear predictors. With multivariate responses, POCRE can construct common components and thus build up latent-variable models for large p small n data.

💡 Research Summary

The paper introduces Penalized Orthogonal‑Components Regression (POCRE), a novel method designed for high‑dimensional, low‑sample‑size (p≫n) problems where traditional regression techniques struggle due to singular covariance matrices and the difficulty of variable selection. The core idea combines two concepts: (1) sequential construction of orthogonal components that maximize the correlation between the response residuals and linear combinations of all predictors, and (2) a sparsity‑inducing penalty applied to the loading vectors of each component via Empirical Bayes Thresholding (EBT).

In the orthogonal‑components framework, the first component ω₁ is the leading eigenvector of cov(Y,X)ᵀcov(Y,X), i.e., the direction in predictor space that best explains the response. After extracting ωⱼ, both the predictor matrix X and the response matrix Y are deflated by removing the projection onto ωⱼ, guaranteeing that the next component is orthogonal to all previous ones. Repeating this process yields a set of mutually orthogonal predictors {ΩᵀX}₁,…,ₗ that can be used in a regression model Y≈∑_{j=1}^l ηⱼQⱼ, where ηⱼ=ΩⱼᵀX.

Sparsity is introduced by reformulating the eigenvector problem as a penalized optimization: minimize ‖M−Mγ αᵀ‖_F² + κ‖γ‖₂² + p_λ(γ) subject to ‖α‖=1, where M is an estimate of cov(Y,X). The ℓ₁‑type penalty p_λ(γ) encourages many entries of γ to be zero. The optimization proceeds by alternating updates of α and γ. The γ‑update reduces to a denoising problem for Z = MᵀMα, solved via Empirical Bayes Thresholding. In EBT each z_i is modeled as μ_i+ε_i with ε_i∼N(0,σ²); μ_i follows a mixture prior with a point mass at zero and a quasi‑Cauchy slab. The mixing weight w and σ are estimated from the data, and the posterior median provides an adaptive thresholding rule that automatically adapts to the underlying sparsity level.

The algorithm starts with γ as the leading eigenvector of XᵀY YᵀX, computes α = (XᵀY YᵀX γ)/‖XᵀY YᵀX γ‖, estimates σ̂ robustly using the median absolute deviation, applies the EBT shrinkage to obtain a new γ, and iterates until convergence. The final loading ωⱼ = γ/‖γ‖ defines the j‑th orthogonal component, ηⱼ = Xⱼ ωⱼ, and the predictor matrix is updated as X_{j+1}=X_j−ηⱼPⱼ with Pⱼ = ηⱼᵀX_j/(ηⱼᵀηⱼ).

Simulation studies cover five scenarios: (1) highly correlated predictors, (2) moderately correlated predictors, (3) clustered predictors, (4) measurement‑error predictors, and (5) a latent‑variable model with multivariate responses. In each case p=1000 and n=50 or 100. POCRE is compared against LASSO, Elastic Net, Ridge regression, and Partial Least Squares (PLS). Performance is evaluated by prediction loss (expected squared error adjusted for variance) and false discovery rate (FDR). Results show that POCRE consistently yields lower loss and markedly reduced FDR in settings with strong correlation or clustering, outperforming Elastic Net and matching or surpassing LASSO. Ridge and PLS, which use all predictors, exhibit large losses and cannot be assessed for FDR. In the multivariate latent‑variable case, POCRE’s ability to share common components across responses provides a compact latent‑variable representation while retaining predictive accuracy.

Key contributions of the work are: (i) a unified orthogonal‑components regression framework that avoids matrix inversion and is computationally cheap for p≫n; (ii) a data‑driven sparsity mechanism based on empirical Bayes that automatically adapts to the unknown sparsity pattern; (iii) demonstrated grouping of highly correlated variables without explicit group penalties; and (iv) extension to multivariate responses via common components, enabling latent‑variable modeling.

Limitations include the linearity assumption, potential bias when predictors are strongly dependent (since EBT assumes approximate independence), and sensitivity to the tuning parameter λ, which must be selected by cross‑validation. Future directions suggested are nonlinear extensions via kernel methods, incorporation of group‑wise weights to blend with group‑lasso ideas, and scalable parallel implementations for massive datasets. Overall, POCRE offers a promising, theoretically grounded, and practically efficient tool for high‑dimensional regression with built‑in variable selection and grouping capabilities.