From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamics

In this paper, we derive an accelerated continuous-time formulation of Adam by modeling it as a second-order integro-differential dynamical system. We relate this inertial nonlocal model to an existing first-order nonlocal Adam flow through an $α$-refinement limit, and we provide Lyapunov-based stability and convergence analyses. We also introduce an Adam-inspired nonlocal Lagrangian formulation, offering a variational viewpoint. Numerical simulations on Rosenbrock-type examples show agreement between the proposed dynamics and discrete Adam.

💡 Research Summary

This paper presents a novel continuous‑time interpretation of the Adam optimizer by modeling it as a second‑order integro‑differential equation (IDE) with nonlocal (memory) terms. Starting from the discrete Adam update, the authors replace the exponential moving averages of gradients and squared gradients with causal convolution kernels that arise naturally in the limit of vanishing step size (α → 0). The resulting dynamics take the form

\