On the conditions used to prove oracle results for the Lasso

Oracle inequalities and variable selection properties for the Lasso in linear models have been established under a variety of different assumptions on the design matrix. We show in this paper how the different conditions and concepts relate to each other. The restricted eigenvalue condition (Bickel et al., 2009) or the slightly weaker compatibility condition (van de Geer, 2007) are sufficient for oracle results. We argue that both these conditions allow for a fairly general class of design matrices. Hence, optimality of the Lasso for prediction and estimation holds for more general situations than what it appears from coherence (Bunea et al, 2007b,c) or restricted isometry (Candes and Tao, 2005) assumptions.

💡 Research Summary

The paper provides a comprehensive synthesis of the various design‑matrix conditions that have been used to establish oracle‑type results for the Lasso in high‑dimensional linear regression. It begins by recalling the notion of an oracle inequality: the risk of the Lasso estimator is bounded by a constant multiple of the risk that would be achieved by an “oracle” that knows the true sparse support in advance. Such inequalities, together with variable‑selection consistency, are the gold standard for theoretical guarantees of the Lasso.

Historically, two families of conditions have dominated the literature. The first, borrowed from compressed sensing, is the Restricted Isometry Property (RIP) introduced by Candès and Tao (2005). RIP requires that for every s‑sparse vector h the quadratic form of the design matrix satisfies (1‑δ)‖h‖₂² ≤ ‖Xh‖₂²/n ≤ (1+δ)‖h‖₂² with a small δ. The second family is based on coherence, i.e., the maximum absolute inner product between any two columns of X, as used by Bunea et al. (2007). Both RIP and coherence impose very strong restrictions: they essentially demand that the columns of X be almost orthogonal, a condition rarely met in practice, especially when variables are highly correlated (e.g., genomics, image processing, finance).

The authors shift the focus to two milder conditions that have emerged from the statistical literature: the Restricted Eigenvalue (RE) condition of Bickel, Ritov, and Tsybakov (2009) and the Compatibility condition of van de Geer (2007). The RE condition states that there exists a constant κ>0 such that for all vectors h lying in a cone defined by the sparsity pattern S (|S|=s) we have
‖Xh‖₂²/n ≥ κ‖h‖₂².
The Compatibility condition replaces the ℓ₂‑norm on the right‑hand side with an ℓ₁‑norm scaled by √s, i.e., there exists L>0 such that for all h in the same cone,
‖h_S‖₁ ≤ L√s‖Xh‖₂/√n.
The paper proves that the RE condition implies the Compatibility condition (with L = √s/κ), while the converse does not hold without additional regularity (e.g., column‑wise ℓ₂ normalisation). This hierarchy shows that Compatibility is strictly weaker, yet still sufficient for oracle results.

A central technical contribution is the “cone condition” argument. By analysing the optimality conditions of the Lasso, the authors show that the estimation error Δ = β̂ – β⁰ necessarily belongs to a cone determined by the true support S. Within this cone, the RE or Compatibility constants provide lower bounds on the prediction norm ‖XΔ‖₂/√n, which in turn translate into upper bounds on both the ℓ₁‑error and the prediction error. Concretely, under the RE condition with constant κ and a regularisation parameter λ ≥ 2‖Xᵀε‖_∞/n, the following oracle inequalities hold:
‖β̂ – β⁰‖₁ ≤ (4λs)/κ,
‖X(β̂ – β⁰)‖₂²/n ≤ (4λ²s)/κ.
These bounds match the rates obtained under RIP, but the assumptions are far less restrictive.

The paper also discusses why coherence‑based assumptions are overly conservative. Coherence μ requires that the maximum column correlation be small, which forces the design to be nearly orthogonal. In contrast, RE and Compatibility only need a positive eigenvalue on the subspace spanned by the true sparse support; they tolerate arbitrarily large correlations among columns outside that subspace. This flexibility is illustrated with several concrete matrix families: (i) random Gaussian and sub‑Gaussian designs, for which concentration results guarantee κ>0 with high probability; (ii) autoregressive (AR(1)) designs, where columns exhibit strong within‑series correlation yet the RE constant remains bounded away from zero provided the sample size n grows faster than s·log p; (iii) block‑structured designs, where high intra‑block correlation is allowed as long as blocks are sufficiently separated. In all these examples, the coherence can be close to 1, but the RE/Compatibility conditions still hold, demonstrating their broader applicability.

The authors further argue that the RE condition is essentially the weakest known condition that still yields sharp oracle inequalities for the Lasso. While RIP is sufficient, it is not necessary; many matrices that violate RIP nevertheless satisfy RE. Moreover, the Compatibility condition, being even weaker, can be verified in settings where estimating κ directly is difficult, because it only requires bounding an ℓ₁‑ℓ₂ ratio.

Finally, the paper acknowledges practical challenges. Estimating κ or L from data is non‑trivial, and the paper suggests possible approaches such as sample‑splitting, bootstrap calibration, or using restricted eigenvalue estimators based on convex optimisation. It also points to extensions beyond the linear model, including generalized linear models and non‑convex penalties, where analogous RE‑type conditions are being investigated.

In summary, the article unifies a fragmented body of literature by showing that the Restricted Eigenvalue and Compatibility conditions form a natural, minimally restrictive framework for proving oracle results for the Lasso. By demonstrating that these conditions encompass a much larger class of design matrices than coherence or RIP, the authors broaden the theoretical foundations of the Lasso and reinforce its relevance for modern high‑dimensional applications where strong collinearity is the norm rather than the exception.