Geometric Prior-Guided Federated Prompt Calibration

Geometric Prior-Guided Federated Prompt Calibration Fei Luo1, Ziwei Zhao2, Mingxuan Wang3, Duoyang Li4, Zhe Qian5, Jiayi Tuo6, Chenyue Zhou7 Yanbiao Ma3∗, 1Jishou University, 2Technical University of Munich, 3Renmin University of China, 4Northwestern Polytechnical University, 5South China Agricultural University, 6University of Science and Technology of China, 7Nanjing University of Aeronautics and Astronautics Abstract Federated Prompt Learning (FPL) offers a parameter-efficient solution for collab- oratively training large models, but its performance is severely hindered by data heterogeneity, which causes locally trained prompts to become biased. Existing methods, focusing on aggregation or regularization, fail to address this root cause of local training bias. To this end, we propose Geometry-Guided Text Prompt Cali- bration (GGTPC), a novel framework that directly corrects this bias by providing clients with a global geometric prior. This prior, representing the shape of the global data distribution derived from the covariance matrix, is reconstructed on the server in a privacy-preserving manner. Clients then use a novel Geometry-Prior Calibration Layer (GPCL) to align their local feature distributions with this global prior during training. Extensive experiments show GGTPC’s effectiveness. On the label-skewed CIFAR-100 dataset (β=0.1), it outperforms the state-of-the-art by 2.15%. Under extreme skew (β=0.01), it improves upon the baseline by 9.17%. Furthermore, as a plug-and-play module on the domain-skewed Office-Home dataset, it boosts FedAvg’s performance by 4.60%. These results demonstrate that GGTPC effectively mitigates data heterogeneity by correcting the fundamental local training bias, serving as a versatile module to enhance various FL algorithms. 1 Introduction Federated Learning (FL) has garnered significant attention from both academia and industry for its ability to collaboratively train shared global models across multiple clients while preserving local data privacy[1, 2, 3, 4, 5, 6, 7]. In recent years, visual foundation models like CLIP[8], BLIP[9] and BLIP2[10] have achieved breakthroughs across numerous visual tasks due to their robust generalization capabilities. Consequently, securely and efficiently integrating such models into federated learning frameworks has emerged as a critical research direction[11, 12]. However, directly fine-tuning these large scale models on clients incurs substantial computational and communication overhead. To address this, parameter-efficient alternatives like Federated Prompt Learning (FPL) have emerged [12, 13, 14]. This approach significantly boosts training efficiency by communicating and aggregating only lightweight prompt vectors across clients, rapidly becoming a research hotspot in parameter-efficient federated learning. Recent surveys and domain deployments further illustrate FL’s breadth across robust training, continual on-device learning, sequential recommendation, oncology data sharing, and scientific collaboration ecosystems[15, 16, 17, 18, 19]. Despite its promising prospects, the core challenge of federated prompt learning is the widespread data heterogeneity among clients[20, 21, 22, 23, 24]. This stems from the intrinsic mechanism of prompt learning: the method aims to find optimal prompts that center text embeddings precisely ∗Corresponding author. Email: ybma1998@ruc.edu.cn Preprint. Under review. arXiv:2512.07208v1 [cs.LG] 8 Dec 2025 around the global visual feature center. In a federated setting, however, each client’s data distribution is often biased (e.g. containing only partial categories or exhibiting severe class imbalance), resulting in a significant discrepancy between local data and the ideal global distribution[25]. When clients train prompts based solely on their own biased local data, the resulting text prompts inevitably deviate from the true global visual center. This performance degradation, caused by aggregating locally biased updates, severely limits the practical application of federated prompt learning[26, 27]. To address the aforementioned issues, we propose a direct solution: guiding clients towards unbiased local calibration by providing them with prior knowledge of the global data distribution in a privacy- preserving manner. This avoids optimization within the limited scope of local data. The core idea is to leverage the geometric properties of the embedding distribution to efficiently quantify and transmit this global prior information. Specifically, we define the geometric shape of each category in the embedding space as its primary data distribution direction (the eigenvector of the covariance matrix) and the extent to which it spreads across directions (the corresponding eigenvalue). This geometric prior accurately captures the intrinsic structure of the global data and inherently protects privacy, as its transmission involves no raw data. Based on the aforementioned core principles, we propose a nov

…(Full text truncated)…

This content is AI-processed based on ArXiv data.