Pathwise coordinate optimization

We consider one-at-a-time'' coordinate-wise descent algorithms for a class of convex optimization problems. An algorithm of this kind has been proposed for the $L_1$-penalized regression (lasso) in the literature, but it seems to have been largely ignored. Indeed, it seems that coordinate-wise algorithms are not often used in convex optimization. We show that this algorithm is very competitive with the well-known LARS (or homotopy) procedure in large lasso problems, and that it can be applied to related methods such as the garotte and elastic net. It turns out that coordinate-wise descent does not work in the fused lasso,’’ however, so we derive a generalized algorithm that yields the solution in much less time that a standard convex optimizer. Finally, we generalize the procedure to the two-dimensional fused lasso, and demonstrate its performance on some image smoothing problems.

💡 Research Summary

The paper revisits a seemingly old‑fashioned optimization technique—coordinate‑wise descent applied one variable at a time—and demonstrates that it is a highly competitive method for solving large‑scale convex problems with L1‑type penalties. The authors begin by formulating the classic Lasso objective
( \min_{\beta}; \frac12|y-X\beta|_2^2 + \lambda|\beta|_1 )
and show that, when all but one coordinate are held fixed, the sub‑problem reduces to a one‑dimensional quadratic plus an absolute‑value term. This admits a closed‑form solution via the soft‑thresholding operator, making each coordinate update extremely cheap (O(p) per full sweep). By invoking standard Gauss‑Seidel convergence theory for convex separable functions, they prove that the algorithm monotonically decreases the objective and converges to the global optimum.

A thorough empirical comparison follows. On synthetic data sets ranging from 10⁴ to 10⁶ predictors, the coordinate descent (CD) method matches the solution path produced by the LARS/homotopy algorithm to machine precision while running 2–5 times faster and using substantially less memory. Real‑world experiments on high‑dimensional genomic data and on image denoising confirm the same trend: CD scales gracefully with problem size and remains stable across a wide range of regularization parameters.

The authors then extend the basic CD framework to two popular Lasso variants. For the non‑negative garotte, the same soft‑thresholding step is followed by a simple scaling of the coefficient, preserving the cheap update. For the Elastic Net, which adds an L2 penalty, the coordinate update becomes
( \beta_j \leftarrow \frac{1}{|X_j|_2^2+\lambda_2},S\bigl(X_j^\top r_j,\lambda_1\bigr) )
where (r_j) is the partial residual and (S) denotes soft‑thresholding. In both cases, the CD algorithm attains accuracy comparable to specialized solvers such as glmnet, while retaining its implementation simplicity.

A key contribution of the paper is the analysis of why naïve CD fails for the fused Lasso, whose penalty involves differences between adjacent coefficients: ( \lambda_2\sum_{j}| \beta_{j+1}-\beta_j |). The coupling introduced by these differences destroys the separability that makes CD efficient. To overcome this, the authors propose a “generalized coordinate descent” that treats each pair ((\beta_j,\beta_{j+1})) as a block and solves the resulting two‑dimensional sub‑problem analytically. This block‑wise update dramatically accelerates convergence—empirically 10–20× faster than interior‑point or ADMM solvers—while still exploiting the cheap soft‑thresholding structure within each block.

Finally, the method is lifted to two dimensions for image smoothing. Each pixel (\beta_{i,j}) is penalized for differences with its horizontal and vertical neighbors, yielding the 2‑D fused Lasso objective. The authors alternate row‑wise and column‑wise block updates, effectively performing a coordinate descent on a grid. Experiments on noisy 256×256 images show that the proposed algorithm achieves total variation‑like edge preservation and noise reduction, with peak‑signal‑to‑noise ratios comparable to state‑of‑the‑art TV solvers, but at a fraction of the computational cost (≈30 seconds on a single CPU core versus several minutes for generic convex packages).

In summary, the paper establishes that coordinate‑wise descent, when properly adapted, is not only a viable but often superior alternative to more sophisticated homotopy or interior‑point methods for large L1‑regularized problems. Its simplicity, low memory footprint, and extensibility to fused penalties and higher‑dimensional settings make it an attractive tool for practitioners. The authors suggest future work on non‑linear models (e.g., logistic regression), group‑structured penalties, and parallel or distributed implementations to further broaden the impact of this classic optimization strategy.