ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations

Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning, due to its reduced number of trainable parameters and lower memory requirements enabled by Burer-Monteiro factorization on adaptation matrices. However, classical LoRA training methods treat the low-rank factor matrices individually and optimize them using standard gradient-based algorithms. Such decoupled optimization schemes are theoretically and empirically suboptimal, as they fail to fully exploit the intrinsic structure of the LoRA parameterization. In this work, we propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE) that emulates the gradient flow of full fine-tuning on the balanced manifold. We term this approach ODELoRA. To faithfully track the trajectories of ODELoRA, we adopt well-established and theoretically grounded time-discretization schemes, including Euler and Runge–Kutta methods. Our framework provides a unified ODE-based perspective for understanding and designing LoRA training algorithms. We establish linear convergence of the proposed method under strongly convex objectives for certain discretization schemes under mild conditions, and further extend our analysis to the matrix sensing setting. Moreover, we show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality. Empirical results on matrix sensing tasks confirm the derived linear convergence behavior, and experiments on training physics-informed neural networks further demonstrate the superiority of ODELoRA over existing baselines, especially in the training stability.

💡 Research Summary

The paper introduces ODELoRA, a novel training paradigm for Low‑Rank Adaptation (LoRA) that replaces the conventional decoupled gradient updates of the factor matrices (A) and (B) with a continuous‑time dynamics formulated as an ordinary differential equation (ODE). Traditional LoRA fine‑tuning treats (A) and (B) as independent parameters and applies standard optimizers (SGD, Adam) to each, which ignores the multiplicative structure (W = W_{\text{pt}} + BA) and leads to a mismatch with the gradient flow of the full‑parameter model. This mismatch can cause sub‑optimal performance and instability, especially because the magnitudes of (A) and (B) can become imbalanced.

ODELoRA addresses this by defining a time‑dependent pair ((A(t), B(t))) and solving at each instant the following least‑squares problem:

\