Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent

First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. We observe that the performances of gradient and mirror descent are complementary, so that faster algorithms can be designed by LINEARLY COUPLING the two. We show how to reconstruct Nesterov’s accelerated gradient methods using linear coupling, which gives a cleaner interpretation than Nesterov’s original proofs. We also discuss the power of linear coupling by extending it to many other settings that Nesterov’s methods cannot apply to.

💡 Research Summary

The paper “Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent” presents a unifying framework that combines the two most fundamental first‑order optimization primitives—gradient descent (GD) and mirror descent (MD)—into a single algorithmic scheme called linear coupling. The authors begin by reviewing the classic roles of GD and MD. Gradient descent operates in the primal space: under an L‑smoothness assumption it takes a step that maximally decreases a quadratic upper bound on the objective, guaranteeing a per‑iteration decrease of at least (1/2L)‖∇f(x)‖*². Mirror descent, by contrast, works in the dual space: each queried gradient defines a supporting hyperplane, and MD builds a convex combination of these hyperplanes to produce a lower bound on the optimum. The quality of this lower bound is measured by a regret term that can be regularized using a strongly convex distance‑generating function (DGF) and its associated Bregman divergence.

The key insight of the paper is that GD provides fast progress when gradient norms are large, while MD provides fast progress when gradient norms are small. By coupling the two updates linearly at every iteration, the algorithm can enjoy the best of both worlds. Concretely, at iteration k the method computes a GD step yₖ = Grad(xₖ) and an MD step zₖ = Mirror(xₖ), then forms a convex combination xₖ₊₁ = τ zₖ + (1 − τ) yₖ, where τ∈(0,1) is a parameter chosen to balance the two progress terms. The same gradient ∇f(xₖ₊₁) is used for the next iteration, ensuring that the two sequences remain synchronized.

A simple “thought experiment” shows that if one could decide in advance whether the gradient norm is always above or below a threshold K, then either pure GD or pure MD would achieve an iteration bound of O(L/(K²) + K²/ε²). By selecting K = √(Lε) the bound becomes O(√(L/ε)), which matches the optimal rate of Nesterov’s accelerated gradient method. The linear coupling construction turns this heuristic into a rigorous algorithm: the combined update guarantees that the primal decrease from GD and the dual improvement from MD jointly shrink a potential function at the accelerated rate.

The authors then demonstrate that the linear‑coupling framework is agnostic to the choice of norm. By using a norm‑compatible smoothness definition and a DGF that is 1‑strongly convex with respect to the same norm, the same analysis holds for ℓ₁, ℓ∞, or any other normed space. The method also naturally extends to constrained settings (Q ⊂ ℝⁿ) by replacing Euclidean projections with Bregman projections, and to proximal settings where the objective is f + g with a possibly nonsmooth regularizer g; the GD step becomes a proximal gradient step while the MD step remains unchanged.

Beyond reproducing Nesterov’s accelerated method, the paper showcases several applications where traditional acceleration does not directly apply. For instance, in problems with asymmetric constraints, composite regularizers, or where the smoothness constant varies across coordinates, the linear‑coupling approach still yields accelerated convergence. The authors provide concrete algorithmic variants and theoretical guarantees for these scenarios, illustrating the versatility of the framework.

In summary, the paper offers a conceptually simple yet powerful unification: by linearly coupling a primal (gradient) update with a dual (mirror) update at each iteration, one obtains an accelerated first‑order method that is easier to understand, more modular, and applicable to a broader class of problems than classical Nesterov acceleration. This work deepens our understanding of acceleration as a balance between primal and dual progress, and opens new avenues for designing fast algorithms in modern large‑scale optimization and machine learning.