Learning to Reason in 13 Parameters

Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.

💡 Research Summary

The paper “Learning to Reason in 13 Parameters” introduces TinyLoRA, a novel parameter‑efficient adaptation technique that pushes the limits of low‑rank adapters far beyond what prior work has achieved. Conventional LoRA, even in its smallest rank‑1 configuration, still requires millions of trainable parameters (e.g., ~3 M for Llama‑3‑8B). The authors ask whether such a large number is truly necessary for teaching a language model to reason, especially when reinforcement learning (RL) appears to be far more information‑efficient than supervised fine‑tuning (SFT).

TinyLoRA is built on two key ideas. First, it replaces the trainable r × r matrix R of LoRA‑XS with a low‑dimensional vector v ∈ ℝᵘ that is projected through a set of fixed random tensors Pᵢ (each of size u × r × r). The update rule becomes
 W′ = W + U Σ (∑ᵢ vᵢ Pᵢ) Vᵀ,
where U, Σ, V are the truncated SVD components of the frozen weight matrix W. By choosing u = 1, each adapted module needs only a single scalar. Second, the authors share this scalar across all modules (weight‑tying), so the entire model can be updated with a single trainable parameter. Consequently, the total number of trainable parameters scales as O(n m u / n_tie) and can be reduced to one.

The paper provides a theoretical comparison of the information content delivered by SFT versus RL. SFT learns from full token sequences, forcing the model to store both task‑relevant structure and irrelevant details, which inflates the required capacity. RL, by contrast, repeatedly samples fresh continuations and uses a binary reward signal; the useful information is concentrated in the reward, making the learning signal much denser per bit. This hypothesis predicts that RL should succeed with far fewer parameters than SFT, especially in the ultra‑low‑parameter regime.

Empirical validation uses Qwen2.5‑7B‑Instruct and Llama‑3‑8B families across a suite of math reasoning benchmarks: GSM8K, MATH500, AIME, AMC, and others. The authors evaluate four adaptation strategies (full fine‑tuning, LoRA, LoRA‑XS, TinyLoRA) under both SFT and the Group Relative Policy Optimization (GRPO) RL algorithm. Results are striking: with TinyLoRA and GRPO, training only 13 bf16 parameters (26 bytes) yields 91 % pass@1 on GSM8K, within 5 % of full fine‑tuning. Even with as few as 120 parameters, the method recovers 95 % of the total performance gain. On the harder MATH500 and olympiad‑level datasets, 196 parameters retain 87 % of the absolute improvement, demonstrating that the approach scales to more challenging reasoning tasks. In contrast, SFT with the same tiny adapters barely moves the model beyond the base performance, confirming the superior information density of RL updates.

The authors also explore the relationship between model size and required adapter size. Larger backbones need proportionally smaller updates to reach a given performance threshold, confirming a “scale‑efficiency” trend: an 8 B model can achieve near‑peak performance with just a handful of parameters, while a 3 B model still benefits from sub‑kilobyte adapters.

Implementation challenges are addressed by integrating TinyLoRA into the vLLM inference stack, which only supports LoRA ranks ≥ 4. The authors circumvent this by merging the adapter weights into the base model for training and applying the separate LoRA weights only at inference time, using truncated importance sampling to mitigate the resulting distribution shift.

In summary, TinyLoRA demonstrates that reasoning capabilities can be injected into billion‑parameter language models with an adapter as small as a single scalar, provided the learning algorithm is reinforcement‑based. This breakthrough opens the door to ultra‑lightweight personalization, massive multi‑tenant serving, and efficient on‑device adaptation of large language models, all while preserving most of the performance gains traditionally obtained with far larger parameter budgets.