Graph Representation-based Model Poisoning on the Heterogeneous Internet of Agents

Internet of Agents (IoA) envisions a unified, agent-centric paradigm where heterogeneous large language model (LLM) agents can interconnect and collaborate at scale. Within this paradigm, federated fine-tuning (FFT) serves as a key enabler that allows distributed LLM agents to co-train an intelligent global LLM without centralizing local datasets. However, the FFT-enabled IoA systems remain vulnerable to model poisoning attacks, where adversaries can upload malicious updates to the server to degrade the performance of the aggregated global LLM. This paper proposes a graph representation-based model poisoning (GRMP) attack, which exploits overheard benign updates to construct a feature correlation graph and employs a variational graph autoencoder to capture structural dependencies and generate malicious updates. A novel attack algorithm is developed based on augmented Lagrangian and subgradient descent methods to optimize malicious updates that preserve benign-like statistics while embedding adversarial objectives. Experimental results show that the proposed GRMP attack can substantially decrease accuracy across different LLM models while remaining statistically consistent with benign updates, thereby evading detection by existing defense mechanisms and underscoring a severe threat to the ambitious IoA paradigm.

💡 Research Summary

The paper addresses security vulnerabilities in the emerging Internet of Agents (IoA) paradigm, where heterogeneous large language model (LLM) agents collaboratively fine‑tune a shared backbone through federated fine‑tuning (FFT). In this setting, each agent transmits only low‑rank LoRA updates over a wireless medium, preserving privacy and reducing bandwidth. The authors observe that the broadcast nature of wireless links enables a malicious participant to eavesdrop on benign updates and to craft poisoned updates that are statistically indistinguishable from legitimate ones.

To exploit this, they propose the Graph Representation‑based Model Poisoning (GRMP) attack. The attack proceeds in several stages. First, the adversary collects a set of benign LoRA updates ({\Delta w_i(t)}) and constructs a graph (G = (V, E, F)) where nodes correspond to parameter dimensions, edges encode cosine similarity between dimensions, and the feature matrix (F) stacks the updates of all observed agents. This graph captures the correlation structure inherent in benign updates.

Second, a variational graph auto‑encoder (VGAE) is trained on ((A, F)), where (A) is the adjacency matrix derived from cosine similarities. The VGAE consists of a multi‑layer Graph Convolutional Network (GCN) encoder that maps the graph to a low‑dimensional latent representation (Z), and a decoder that reconstructs the adjacency matrix via a sigmoid of (ZZ^\top). The training objective combines a reconstruction loss with a KL‑divergence regularizer, encouraging the latent space to preserve the structural dependencies of the benign updates.

Third, the attacker uses a Graph Signal Processing (GSP) module. The Laplacian of the original adjacency matrix is eigen‑decomposed to obtain a Graph Fourier Transform (GFT) basis (B). The benign feature matrix is projected onto this basis, yielding spectral coefficients (S = FB). From the VGAE‑generated reconstructed adjacency (\hat{A}), a new Laplacian (\hat{L}) and its eigen‑basis (\hat{B}) are derived. The attacker then reconstructs a feature matrix (\hat{F}=S\hat{B}^\top) and selects one row as the malicious LoRA update (\Delta w’_j(t)).

The malicious update must satisfy a stealth constraint: its Euclidean distance to the current global update (\Delta w_g(t)) must stay below a dynamic threshold (d_T(t)) (the mean distance among benign updates). To enforce this while maximizing the global loss (F(w’_g)), the authors formulate a constrained optimization problem and introduce an augmented Lagrangian
\