Parameter-Efficient Subspace Optimization for LLM Fine-Tuning

This paper develops a new perspective on parameter-efficient fine-tuning (PEFT) for LLMs, inspired by classical subspace minimization. We introduce a unifying framework, Parameter-Efficient Subspace Optimization (PESO), which recovers existing methods such as LoRA and connects them to the principled algorithmic and theoretical foundations of subspace optimization. This connection highlights a natural ``exploration–exploitation’’ view of subspace methods, guiding the design of new algorithms that achieve strong convergence performance while still preserving memory efficiency. We instantiate the framework into a practical algorithm, PESO-LoRA, based on a LoRA-type parameterization. Importantly, we provide convergence guarantees stated in the full-parameter space for the induced update, addressing a key limitation of LoRA-style analyses that only track low-dimensional factors. Empirically, PESO-LoRA improves over strong PEFT baselines on standard fine-tuning benchmarks.

💡 Research Summary

This paper presents a novel perspective on Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs) by drawing inspiration from the classical optimization concept of subspace minimization. The authors introduce a unifying framework called Parameter-Efficient Subspace Optimization (PESO), which reformulates PEFT as an iterative process of solving a sequence of subproblems within carefully chosen, evolving low-dimensional subspaces.

The core idea of PESO is to decompose the high-dimensional fine-tuning problem into manageable parts. At each iteration k, the model weights W_k are represented as the sum of an anchored weight Ŵ_k (accumulating progress from past subspaces) and a mapping M_k(ξ_k) that projects low-dimensional parameters ξ_k onto the current subspace S_k. The framework operates through three complementary operations: 1) Exploration: Periodically updating the subspace S_k (via M_k) and the anchor Ŵ_k using information like the full gradient. 2) Exploitation: Performing multiple steps of a standard optimizer (e.g., Adam) within the current subspace S_k to update ξ_k. 3) Anchor Update: Absorbing the progress made in a subspace back into the anchored weights. The exploration phase can operate in two modes: warm-start, which smoothly evolves the existing subspace, and restart, which resets the subspace based on new information (e.g., SVD of the gradient) and re-initializes ξ_k.

The PESO framework provides a unifying lens through which many existing memory-efficient training methods can be viewed. For instance, standard LoRA corresponds to a PESO instance with a fixed anchor (W_0) and a fixed low-rank mapping (AB), performing only exploitation. Methods like GaLore, which periodically reset the projection subspace, align with the restart strategy. This perspective not only connects disparate PEFT techniques but also guides the design of new algorithms by emphasizing the balance between exploring new subspaces and exploiting the current one.

Guided by this framework, the authors propose a practical instantiation called PESO-LoRA, which employs a LoRA-type parameterization for M_k(ξ_k). A key theoretical contribution is providing convergence guarantees in the full-parameter space for the update induced by PESO-LoRA. This addresses a significant limitation of traditional LoRA-style analyses, which only track convergence in the low-dimensional factor space, and theoretically ensures that the method can bridge the performance gap with full fine-tuning while maintaining parameter efficiency.

Empirically, two variants are evaluated: PESO-LoRA-R (using restart) and PESO-LoRA-T (using warm-start). Extensive experiments on benchmarks including GLUE, mathematical reasoning (MetaMathQA), code generation (HumanEval), and instruction tuning (Alpaca) demonstrate that PESO-LoRA improves over strong PEFT baselines like LoRA, LoRA+, and GaLore. The restart-based variant (PESO-LoRA-R) is particularly effective in tasks where a significant performance gap exists between LoRA and full fine-tuning, showcasing the benefit of periodically exploring new subspaces to escape the limitations of a fixed, low-dimensional representation.

In summary, this work bridges classical optimization theory and modern LLM fine-tuning needs. By framing PEFT as subspace minimization, it offers a principled foundation for designing memory-efficient algorithms that do not sacrifice convergence quality, paving the way for more robust and theoretically grounded parameter-efficient training strategies.