Adaptive Accelerated Gradient Descent Methods for Convex Optimization

This work proposes A$^2$GD, a novel adaptive accelerated gradient descent method for convex and composite optimization. Smoothness and convexity constants are updated via Lyapunov analysis. Inspired by stability analysis in ODE solvers, the method triggers line search only when accumulated perturbations become positive, thereby reducing gradient evaluations while preserving strong convergence guarantees. By integrating adaptive step size and momentum acceleration, A$^2$GD outperforms existing first-order methods across a range of problem settings.

💡 Research Summary

**
The paper introduces Adaptive Accelerated Gradient Descent (A²GD), a first‑order optimization algorithm designed for both smooth strongly‑convex (or merely convex) problems and composite convex problems of the form f(x)=h(x)+g(x) with h smooth and g possibly nonsmooth. The authors begin by highlighting two fundamental shortcomings of classic gradient‑based methods: (1) sensitivity to step‑size selection, especially when a global Lipschitz constant L over‑estimates local curvature, and (2) slow convergence on ill‑conditioned problems where the condition number κ=L/µ is large.

To address these issues, the authors develop a Lyapunov‑based analysis. They define a Lyapunov function Eₖ = f(xₖ)−f(x★) and derive a recursive inequality Eₖ₊₁ ≤ δₖ(Eₖ+bₖ) with δₖ=(1+µ/Lₖ)⁻¹. The term bₖ splits into two components: b₁ₖ = ½Lₖ⁻¹‖∇fₖ₊₁−∇fₖ‖²−D_f(xₖ,xₖ₊₁) and b₂ₖ = −½Lₖ⁻¹‖∇fₖ‖². Classical backtracking line‑search forces b₁ₖ≤0 (ℓ∞ condition) at every iteration, incurring extra gradient and function evaluations. Instead, A²GD introduces an accumulated perturbation pₖ = ∑{i=0}^{k}(∏{j=i}^{k}δⱼ) bᵢ and activates line‑search only when pₖ>0. This ℓ₁‑weighted condition allows many iterations to proceed without any line‑search, especially early on when large gradients make b₂ₖ strongly negative and offset positive b₁ₖ.

Step sizes are set adaptively as αₖ=1/Lₖ, where Lₖ is a locally estimated Lipschitz constant obtained from the Lyapunov analysis. When line‑search is required, the algorithm updates Lₖ via Lₖ←r·Lₖ/v with v=2LₖD_f(xₖ,xₖ₊₁)/‖∇fₖ₊₁−∇fₖ‖², rather than the traditional fixed‑ratio increase. This scaling quickly satisfies the descent condition while keeping the number of backtracking steps bounded by O(log(L/r)).

For acceleration, the authors reinterpret Nesterov’s accelerated gradient as a continuous‑time Hamiltonian system (the HNA‑G flow) and discretize it using an IMEX scheme. The discrete updates involve two variables xₖ and yₖ, with a momentum parameter αₖ chosen as αₖ= q·µₖ/Lₖ so that the first perturbation term (I) in Lemma 3.1 vanishes. The remaining perturbations (II) and (III) are again split into b₁ₖ and b₂ₖ, now measuring violations of the Lipschitz condition and of the strong‑convexity assumption, respectively. When pₖ>0, the algorithm either inflates Lₖ (if b₁ₖ>0) or reduces µₖ (if b₂ₖ>0). The strong‑convexity estimate µₖ is kept above a lower bound ε, which is halved only after certain decay criteria are met (either a relative decrease in the Lyapunov value or a fixed number of iterations). This prevents µₖ from collapsing too quickly, preserving the linear convergence rate.

Theoretical results:

• For µ>0, the Lyapunov sequence satisfies Eₖ ≤ ∏_{i=0}^{k‑1}(1+µ/Lᵢ)⁻¹ E₀ ≤ (1+µ/(c_r L))⁻ᵏ E₀, where c_r≥1 depends on the backtracking scaling factor. Hence A²GD achieves accelerated linear convergence, improving over the classic rate (1+µ/L)⁻ᵏ by a constant factor.
• For µ=0, the method reduces to an accelerated scheme with the optimal O(1/k²) rate, matching Nesterov’s bound.

Empirical evaluation: The authors test A²GD on logistic regression (a smooth convex problem) and on a composite ℓ₁‑ℓ₂ regularized problem. Compared to Barzilai‑Borwein GD, standard NAG, backtracking NAG, and recent line‑search‑free adaptive methods, A²GD requires far fewer line‑search activations (often less than ten, mostly in the early iterations) and reduces total gradient evaluations by roughly 20‑30 %. Convergence plots show that the Lyapunov values decay faster than the theoretical exponential bound, especially when the condition number is large (κ≈10⁴).

Limitations and future work: The current analysis relies on convexity; extending the framework to non‑convex objectives remains open, though preliminary experiments suggest the method can still succeed when iterates enter locally convex basins. A theoretical upper bound on the number of line‑search triggers is not yet established, though empirical counts are modest. Finally, a stochastic variant of A²GD—potentially bridging the gap to Adam‑style optimizers—has not been developed and is proposed as future research.

In summary, A²GD blends Lyapunov‑driven adaptive step‑size estimation with a novel “perturbation‑based” line‑search activation rule and momentum acceleration. This combination yields a method that retains the fast theoretical rates of accelerated gradient methods while dramatically cutting the overhead associated with traditional backtracking line‑search, as demonstrated both analytically and experimentally.