R-MTLLMF: Resilient Multi-Task Large Language Model Fusion at the Wireless Edge

Multi-task large language models (MTLLMs) are important for many applications at the wireless edge, where users demand specialized models to handle multiple tasks efficiently. However, training MTLLMs is complex and exhaustive, particularly when tasks are subject to change. Recently, the concept of model fusion via task vectors has emerged as an efficient approach for combining fine-tuning parameters to produce an MTLLM. In this paper, the problem of enabling edge users to collaboratively craft such MTLMs via tasks vectors is studied, under the assumption of worst-case adversarial attacks. To this end, first the influence of adversarial noise to multi-task model fusion is investigated and a relationship between the so-called weight disentanglement error and the mean squared error (MSE) is derived. Using hypothesis testing, it is directly shown that the MSE increases interference between task vectors, thereby rendering model fusion ineffective. Then, a novel resilient MTLLM fusion (R-MTLLMF) is proposed, which leverages insights about the LLM architecture and fine-tuning process to safeguard task vector aggregation under adversarial noise by realigning the MTLLM. The proposed R-MTLLMF is then compared for both worst-case and ideal transmission scenarios to study the impact of the wireless channel. Extensive model fusion experiments with vision LLMs demonstrate R-MTLLMF’s effectiveness, achieving close-to-baseline performance across eight different tasks in ideal noise scenarios and significantly outperforming unprotected model fusion in worst-case scenarios. The results further advocate for additional physical layer protection for a holistic approach to resilience, from both a wireless and LLM perspective.

💡 Research Summary

The paper addresses the emerging need for multi‑task large language models (MTLLMs) at the wireless edge, where devices must quickly adapt to a variety of tasks without the heavy communication and computation overhead of traditional federated learning. The authors build on the recent “task‑vector” paradigm, in which each edge device fine‑tunes a shared pre‑trained LLM on its own data, computes the difference between the fine‑tuned and base parameters (the task vector), and sends this compact representation to a central server. The server aggregates the received vectors (typically by averaging with a scaling factor) to obtain a unified MTLLM.

While elegant, this approach assumes reliable transmission of the task vectors. In realistic wireless environments, especially under worst‑case adversarial attacks, the transmitted symbols can be corrupted by carefully crafted noise. The authors first model the uplink as a MIMO multiple‑access channel with MMSE equalization and derive per‑user mean‑squared error (MSE) µ_q and the total sum MSE Σ_q µ_q. They then connect the MSE to the model‑level metric “weight disentanglement error” (WDE), which quantifies cross‑task interference when multiple task vectors are applied simultaneously. By linearizing the logits of the MTLLM around the clean parameters, they show that the deviation in the logits is proportional to the aggregated error term ε_q, which in turn is directly linked to the wireless MSE. Using a hypothesis‑testing formulation (null hypothesis: noise does not significantly alter predictions), they derive a detection threshold T that depends on the variance of the test statistic, which itself scales with Σ_q µ_q. Consequently, high MSE leads to a higher probability of rejecting the null hypothesis, meaning that task interference becomes statistically significant and model performance degrades.

To characterize the most damaging attacks, the paper formulates two saddle‑point optimization problems. The first (P1) minimizes the sum‑rate of all users by choosing a noise covariance matrix C_z under a total power constraint P_N; the second (P2) minimizes the rate of the strongest user, again by shaping C_z. Closed‑form solutions are obtained via eigen‑decomposition of the channel Gram matrix HPH^H, yielding noise that aligns with the dominant eigen‑vectors (for P1) or with the strongest user’s channel (for P2). These worst‑case noise designs illustrate that an intelligent jammer can render conventional ARQ or retransmission schemes ineffective, forcing the system to rely on higher‑layer resilience.

Motivated by this vulnerability, the authors propose R‑MTLLMF, an AI‑driven resilience framework consisting of two complementary modules:

1. Task‑Vector Aggregation with Frozen LLM Parameters – After aggregating the noisy task vectors, the base model’s position, patch, and class embeddings are restored (or frozen) before inference. Since fine‑tuning typically modifies only the self‑attention layers, preserving the embeddings protects the core token representations from being distorted by noise‑induced parameter drift.

2. Few‑Shot Realignment – To correct residual perturbations in the self‑attention weights, a brief few‑shot fine‑tuning step is performed on a small, representative subset of data (≈10 samples per class). This step can be executed at the edge server using either locally stored episodic memory or publicly available data, thereby respecting privacy constraints. The realignment leverages the observation that moderate noise does not drastically change the direction of the task vectors, so a lightweight adaptation suffices to restore the model’s reasoning capabilities.

The framework is evaluated on vision‑transformer based VIT‑LLMs across eight diverse visual tasks (classification, detection, segmentation, etc.). In the ideal (noise‑free) scenario, both the baseline MTMF and R‑MTLLMF achieve performance within 0.5 % of the original single‑task fine‑tuned models, confirming that the aggregation and scaling strategy does not sacrifice accuracy. Under the worst‑case adversarial noise (as defined by P1 and P2), the unprotected MTMF suffers a dramatic increase in WDE (≈2.3×) and a drop in accuracy to below 30 %. In contrast, R‑MTLLMF reduces WDE to roughly 0.8× of the baseline and recovers average accuracy to about 78 %, demonstrating substantial resilience. The authors also discuss how additional physical‑layer defenses (e.g., spectrum spreading, jammer detection) can be combined with R‑MTLLMF for a holistic protection strategy.

In summary, the paper makes three key contributions: (i) a rigorous analytical link between wireless MSE and model‑level task interference, (ii) a worst‑case noise covariance design that quantifies the limits of purely physical‑layer defenses, and (iii) a practical, AI‑centric resilience mechanism that leverages LLM architectural insights (embedding freezing) and lightweight few‑shot adaptation. The work paves the way for robust, on‑demand multi‑task LLM services in future 6G networks, where both the communication channel and the AI model must be jointly hardened against sophisticated attacks. Future directions include extending the approach to other parameter‑efficient fine‑tuning methods, reducing latency of the few‑shot realignment, and co‑optimizing physical‑ and AI‑layer defenses in an end‑to‑end design.