Sparse Empirical Bayes Analysis (SEBA)

We consider a joint processing of $n$ independent sparse regression problems. Each is based on a sample $(y_{i1},x_{i1})…,(y_{im},x_{im})$ of $m$ \iid observations from $y_{i1}=x_{i1}\t\beta_i+\eps_{i1}$, $y_{i1}\in \R$, $x_{i 1}\in\R^p$, $i=1,…,n$, and $\eps_{i1}\dist N(0,\sig^2)$, say. $p$ is large enough so that the empirical risk minimizer is not consistent. We consider three possible extensions of the lasso estimator to deal with this problem, the lassoes, the group lasso and the RING lasso, each utilizing a different assumption how these problems are related. For each estimator we give a Bayesian interpretation, and we present both persistency analysis and non-asymptotic error bounds based on restricted eigenvalue - type assumptions.

💡 Research Summary

The paper “Sparse Empirical Bayes Analysis (SEBA)” tackles the problem of jointly estimating a collection of n independent high‑dimensional sparse linear regression models, each based on m i.i.d. observations (y_{ij}, x_{ij}) with p≫m. In this regime the ordinary empirical risk minimizer and the standard lasso are inconsistent because the design matrix does not satisfy the classical conditions required for reliable variable selection. The authors propose three extensions of the lasso that exploit different types of relationships among the n problems: (i) “lassoes”, (ii) the group lasso, and (iii) the RING lasso (Rotation‑Invariant Group lasso). For each method they give a clear Bayesian interpretation, derive persistency results, and obtain non‑asymptotic error bounds under restricted eigenvalue‑type (RE) assumptions.

1. Lassoes.
The lassoes estimator augments the usual ℓ₁ penalty on each coefficient vector β_i with an additional ℓ₁ penalty on the entire coefficient matrix B =