Implicit regularization of normalized gradient descent

How to find flat minima? We propose running normalized gradient descent, usually reserved for nonsmooth optimization, with sufficiently slowly diminishing step sizes. This induces implicit regularization towards flat minima if an appropriate Lyapunov functions exists in the gradient dynamics. Our analysis shows that implicit regularization is intrinsically a question of nonsmooth analysis, for which we deploy the full power of variational analysis and stratification theory.

💡 Research Summary

The paper investigates how normalized gradient descent (NGD), a classic algorithm originally proposed for nonsmooth optimization, can be harnessed to achieve implicit regularization toward flat minima when combined with a slowly diminishing step‑size schedule. The authors begin by recalling that NGD updates the iterate as
  x_{k+1}=x_k−α_k ∇f(x_k)/‖∇f(x_k)‖,
provided the gradient is non‑zero, and that convergence is guaranteed under the classical Robbins‑Monro conditions ∑α_k=∞ and ∑α_k²<∞. The novelty lies in interpreting NGD as a discretization of the differential inclusion ˙x∈b∇f(x), where b∇f is a set‑valued “normalized sub‑differential” defined to be the unit‑norm direction of the gradient when it exists and the closed unit ball otherwise. This mapping is upper‑semicontinuous with non‑empty compact convex values for locally Lipschitz functions, a property that enables existence of continuous‑time trajectories and boundedness of maximal solutions.

The core theoretical contribution is the introduction of a p‑d‑Lyapunov function g:ℝⁿ→ℝ. A function g is called a p‑d‑Lyapunov function for the discrete dynamics if for every iteration
  g(x_{k+1})−g(x_k) ≤ −ω α_k^p,
with ω>0 and p∈