Momentum LMS Theory beyond Stationarity: Stability, Tracking, and Regret

In large-scale data processing scenarios, data often arrive in sequential streams generated by complex systems that exhibit drifting distributions and time-varying system parameters. This nonstationarity challenges theoretical analysis, as it violates classical assumptions of i.i.d. (independent and identically distributed) samples, necessitating algorithms capable of real-time updates without expensive retraining. An effective approach should process each sample in a single pass, while maintaining computational and memory complexities independent of the data stream length. Motivated by these challenges, this paper investigates the Momentum Least Mean Squares (MLMS) algorithm as an adaptive identification tool, leveraging its computational simplicity and online processing capabilities. Theoretically, we derive tracking performance and regret bounds for the MLMS in time-varying stochastic linear systems under various practical conditions. Unlike classical LMS, whose stability can be characterized by first-order random vector difference equations, MLMS introduces an additional dynamical state due to momentum, leading to second-order time-varying random vector difference equations whose stability analysis hinges on more complicated products of random matrices, which poses a substantially challenging problem to resolve. Experiments on synthetic and real-world data streams demonstrate that MLMS achieves rapid adaptation and robust tracking, in agreement with our theoretical results especially in nonstationary settings, highlighting its promise for modern streaming and online learning applications.

💡 Research Summary

The paper addresses the challenge of online adaptive filtering in non‑stationary environments where both the input data distribution and the underlying system parameters evolve over time. While the classical Least Mean Squares (LMS) algorithm has been extensively studied under stationary or slowly varying assumptions, its momentum‑augmented counterpart (MLMS) introduces a second dynamical state, making the analysis substantially more complex. The authors first formulate a time‑varying linear regression model y_{k+1}=φ_k^⊤θ_k+v_{k+1} and adopt a stochastic persistence‑of‑excitation condition that does not require independence or stationarity of the regressor sequence.

The MLMS update combines an adaptive step‑size α_k=μ/(δ+‖φ_k‖²) with a momentum term β(θ̂_k−θ̂_{k−1}), where β is scaled as C_β μ^κ (C_β∈(0,1], κ>1). By stacking the current and previous estimation errors, the authors rewrite the recursion as a first‑order random matrix difference equation Z_{k+1} = (I_0−\bar A_k) Z_k +