Collaborative and Efficient Fine-tuning: Leveraging Task Similarity

Adaptability has been regarded as a central feature in the foundation models, enabling them to effectively acclimate to unseen downstream tasks. Parameter-efficient fine-tuning methods such as celebrated LoRA facilitate efficient adaptation of large foundation models using labeled, high-quality and generally scarce task data. To mitigate data scarcity in fine-tuning of foundation models, we propose to leverage task similarity across multiple downstream users. Intuitively, users with similar tasks must be able to assist each other in boosting the effective fine-tuning data size. We propose Collaborative Low-Rank Adaptation, or CoLoRA, which exploits task similarity to collaboratively and efficiently fine-tune personalized foundation models. The main idea in CoLoRA is to train one shared adapter capturing underlying task similarities across all tasks, and personalized adapters tailored to user-specific tasks. We theoretically study CoLoRA on heterogeneous linear regression and provide provable guarantees for ground truth recovery. We also conduct several natural language experiments with varying task similarity, which further demonstrate that when trained together with similar tasks, individual performances are significantly boosted.

💡 Research Summary

The paper tackles the data‑scarcity problem that plagues fine‑tuning of large foundation models (FMs) by exploiting similarity among downstream tasks. While parameter‑efficient fine‑tuning (PEFT) methods such as LoRA reduce the number of trainable parameters by learning a low‑rank update (ΔW = BA), they still require a separate pair of matrices for each task, leading to a total parameter count that scales linearly with the number of users (k) (i.e., (O(krd))). The authors observe that tasks which are semantically related often produce LoRA adapters whose column (and row) subspaces overlap significantly. To quantify this, they introduce a column‑subspace similarity metric (\text{sim}_c(T_i,T_j)=\frac{1}{\sqrt r}|U_i^\top U_j|_F), where (U_i) is an orthonormal basis for the column space of (B_iA_i). Empirical measurements on six synthetic list‑processing tasks confirm that intuitively similar tasks exhibit high similarity scores.

Motivated by this observation, the authors propose Collaborative Low‑Rank Adaptation (CoLoRA). CoLoRA decomposes each task’s adapter into a shared low‑rank component (B A) and a task‑specific square matrix (\Lambda_i\in\mathbb R^{r\times r}), yielding the effective update (ΔW_i = B\Lambda_i A). The optimization problem becomes

\