Efficient Hyper-Parameter Search for LoRA via Language-aided Bayesian Optimization

Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) enables resource-efficient personalization or specialization, but it comes at the expense of additional hyperparameter tuning. Although LoRA makes fine-tuning efficient, it is highly sensitive to the choice of hyperparameters, and exhaustive hyperparameter search is still computationally very demanding. To address these challenges, we propose a framework that integrates the domain knowledge of pre-trained LLMs into Bayesian Optimization (BO) to efficiently search for LoRA hyperparameters. To leverage the informed knowledge of LLMs, we repurpose LLMs as a discrete-to-continuous mapping to link the hyperparameters and their domain knowledge with a continuous vector space, where BO is conducted. We design and control the mapping by language prompting, where we provide a domain-aware textual prompt describing the relationships among hyperparameters and their respective roles; thereby, we explicitly inject domain knowledge about LoRA into the LLM in natural language. Also, we model the residual information that is hard to linguistically describe in the prompt with an additional learnable token. This aids BO to sample more high-performing hyperparameters. In addition, by leveraging the observation of the strong correlation between the respective performance obtained from full and subset training datasets in LoRA training regimes, we introduce proxy training and evaluation with a data subset. This further increases the efficiency of our method. We demonstrate that our hyperparameter found with only about 30 iterations achieves more than 20% performance improvement over standard hyperparameters found from about 45,000 combinations.

💡 Research Summary

The paper tackles the costly hyper‑parameter search problem inherent to Low‑Rank Adaptation (LoRA) fine‑tuning of large language models (LLMs). While LoRA dramatically reduces the computational burden of full model fine‑tuning, its performance remains highly sensitive to a set of discrete hyper‑parameters such as rank r, scaling factor α, learning rate, batch size, and dropout rate. Exhaustive grid search over these variables quickly becomes infeasible because the search space grows combinatorially and each evaluation requires expensive model training.

To address this, the authors propose a novel framework that fuses domain knowledge from a pre‑trained LLM with Bayesian Optimization (BO). The key idea is to treat the LLM as a discrete‑to‑continuous mapper: each hyper‑parameter configuration is first rendered as a structured natural‑language template containing the parameter name, value, and a brief explanation of its role and interactions (e.g., “rank r and scaling factor α are usually set proportionally”). This template, together with a learnable token ψ, is fed into a frozen LLM. The LLM produces contextual token embeddings, which are then passed through a trainable projection layer P(·;θ) to obtain a continuous vector z that serves as the input to a Gaussian‑Process surrogate model.

The learnable token captures residual domain information that is difficult to express in the prompt, while the projection layer adapts the raw LLM embeddings to a space that is more amenable to BO. The surrogate model is trained jointly with ψ and θ by maximizing the marginal log‑likelihood of observed performance values y paired with embeddings z. An acquisition function (e.g., Expected Improvement) then selects the next hyper‑parameter configuration to evaluate.

Because full‑dataset training is still expensive, the authors introduce a proxy training evaluation: they fine‑tune LoRA on a small subset of the data (e.g., 10 % of the training set) and use the resulting validation score as an inexpensive proxy for the full‑dataset performance. Empirically, they demonstrate a strong correlation between proxy and full scores, allowing the BO loop to run with dramatically reduced cost per iteration.

Experiments span multiple base LLMs (LLaMA‑7B, Falcon‑40B), several LoRA variants (DoRA, rsLoRA, PiSSA), and a variety of downstream tasks (text classification, summarization, QA). Compared against a brute‑force grid of roughly 45,000 hyper‑parameter combinations, the proposed method reaches comparable or superior performance after only about 30 BO iterations. This translates to a >20 % relative improvement in final task metrics while cutting total search time by a factor of 5–6 thanks to the proxy evaluation. Ablation studies show that removing the learnable token or the projection layer degrades performance by ~8 %, confirming their contribution. The approach also consistently outperforms standard BO that treats hyper‑parameters as purely continuous or that lacks any domain‑knowledge injection.

The paper acknowledges limitations: the LLM inference cost, the need for expert‑crafted prompts, and the fact that proxy‑training correlation may vary across datasets. Future directions include automated prompt generation, leveraging multimodal LLMs, and theoretical analysis of proxy reliability.

In summary, the work presents the first LLM‑augmented BO framework for LoRA hyper‑parameter optimization, demonstrating that natural‑language domain knowledge, when encoded via a learnable token and projection layer, can dramatically improve sample efficiency and final performance. The methodology is generalizable to other discrete‑continuous optimization problems where rich prior knowledge exists but is difficult to formalize mathematically.