Highly Adaptive Principal Component Regression

The Highly Adaptive Lasso (HAL) is a nonparametric regression method that achieves almost dimension-free convergence rates under minimal smoothness assumptions, but its implementation can be computationally prohibitive in high dimensions due to the large basis matrix it requires. The Highly Adaptive Ridge (HAR) has been proposed as a scalable alternative. Building on both procedures, we introduce the Principal Component based Highly Adaptive Lasso (PCHAL) and Principal Component based Highly Adaptive Ridge (PCHAR). These estimators constitute an outcome-blind dimension reduction which offer substantial gains in computational efficiency and match the empirical performances of HAL and HAR. We also uncover a striking spectral link between the leading principal components of the HAL/HAR Gram operator and a discrete sinusoidal basis, revealing an explicit Fourier-type structure underlying the PC truncation.

💡 Research Summary

The paper addresses a critical computational bottleneck in the Highly Adaptive Lasso (HAL) and its ridge‑penalized counterpart (Highly Adaptive Ridge, HAR), both of which achieve near‑dimension‑free convergence rates for non‑parametric regression under very mild smoothness assumptions. HAL builds a saturated dictionary of indicator (or spline‑like) basis functions defined by all axis‑aligned knots across every non‑empty subset of the d covariates. For a sample of size n this yields p ≈ n·(2ᵈ − 1) basis functions, so the design matrix H∈ℝⁿˣᵖ can be astronomically large, making both the construction of H and the subsequent ℓ₁‑regularized optimization prohibitively expensive. HAR replaces the ℓ₁ penalty with an ℓ₂ (ridge) penalty, allowing the fitted values to be expressed through the n × n Gram matrix K = HHᵀ and a closed‑form Sherman‑Morrison‑Woodbury solution. Although K is much smaller than H, its eigen‑decomposition still costs O(n³) and becomes a new bottleneck when n is large.

The authors propose two new estimators—Principal Component Highly Adaptive Lasso (PCHAL) and Principal Component Highly Adaptive Ridge (PCHAR)—that dramatically reduce computational cost by approximating K with its leading k eigen‑components (k ≪ n). After computing the eigendecomposition K = U D Uᵀ, they retain the top‑k eigenvectors U_k and eigenvalues D_k, yielding a low‑rank approximation K ≈ U_k D_k U_kᵀ. This projects the original regression problem onto a k‑dimensional orthogonal score space.

For PCHAR, the ridge solution becomes β̂ = U_k (D_k + λI)⁻¹ U_kᵀ Y, a simple matrix‑vector multiplication that scales as O(nk²). For PCHAL, the orthogonality of the retained scores turns the ℓ₁‑penalized problem into component‑wise soft‑thresholding: compute the scores z = U_kᵀ Y, then apply β̂_j = sign(z_j)·max(|z_j| − λ, 0). The final prediction is reconstructed by projecting back to the original feature space using the retained eigenvectors. Thus both estimators avoid any iterative optimization and achieve orders‑of‑magnitude speed‑ups.

A striking theoretical contribution is the discovery that the leading eigenvectors of the HAL/HAR Gram operator coincide with a discrete sinusoidal (Fourier) basis. The authors prove that, after sorting the covariate values, the j‑th eigenvector has entries proportional to sin(π j t_i) where t_i denotes the normalized rank of observation i. Consequently, the low‑rank truncation is not an arbitrary dimensionality reduction but a natural frequency‑filtering operation aligned with the data’s ordering structure. This spectral link explains why a modest number of components suffices to capture most of the predictive information.

Algorithmically, the workflow is: (1) construct the full HAL feature map h(·) (the same as for HAL/HAR); (2) compute K = hhᵀ; (3) obtain the top‑k eigenpairs using fast randomized SVD or Lanczos methods; (4) apply the closed‑form ridge or soft‑thresholding formulas; (5) for prediction, compute the low‑dimensional scores and map back to the original space. The authors provide open‑source R and Python packages that include cross‑validation utilities, which reuse the eigenvectors across folds, further reducing overhead.

Empirical studies on synthetic data (d = 10–30, n = 500–5000) and several real‑world datasets demonstrate that PCHAL and PCHAR achieve predictive performance virtually indistinguishable from full HAL and HAR, while reducing runtime by factors ranging from 20× to over 200× and dramatically lowering memory usage. The paper also discusses practical guidelines for choosing k (typically a few dozen components suffice) and the impact of the spectral decay of K on accuracy.

In summary, the paper makes three major contributions: (i) it retains the desirable statistical guarantees of HAL/HAR while delivering a scalable implementation suitable for high‑dimensional, large‑sample problems; (ii) it introduces a principled, outcome‑blind dimension reduction based on principal components that leads to closed‑form estimators; and (iii) it uncovers a deep Fourier‑type structure underlying the HAL/HAR kernel, providing a theoretical justification for low‑rank truncation. The work opens avenues for further extensions such as nonlinear kernels, multi‑response settings, and adaptive selection of k, and the released software makes the methods immediately accessible to practitioners.