TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learning

In federated learning (FL), local personalization of models has received significant attention, yet personalized fine-tuning of foundation models remains a significant challenge. In particular, there is a lack of understanding in the literature on how to fine-tune and personalize foundation models in settings that are heterogeneous across clients not only in data, but also in tasks and modalities. To address this gap, we propose TAP (Two-Stage Adaptive Personalization), which has two key features: (i) leveraging mismatched model architectures between the clients and server to selectively conduct replacement operations when it benefits a client’s local tasks; (ii) engaging in post-FL knowledge distillation for capturing beneficial general knowledge without compromising personalization. In developing TAP, we introduce the first convergence analysis of federated foundation model training at the server under its modality-task pair architecture, and demonstrate that as the number of modality-task pairs increases, its ability to cater to all tasks suffers. Through extensive experiments, we demonstrate the effectiveness of our proposed algorithm across a variety of datasets and tasks in comparison to state-of-the-art federated personalization baselines.

💡 Research Summary

This paper tackles the under‑explored problem of personalizing large foundation models in federated learning (FL) when clients differ not only in data distribution but also in the modalities they handle and the tasks they need to solve. Existing personalized FL (PFL) methods typically assume a homogeneous setting where every client shares the same model architecture, modality, and task, which is unrealistic for real‑world applications such as healthcare (image scans vs. text reports) or industrial IoT (acoustic vs. temperature sensors). To bridge this gap, the authors propose TAP (Two‑Stage Adaptive Personalization), a framework that operates in two complementary stages.

Stage 1 – Adaptive Parameter Replacement.
Each client maintains two parallel models: (i) f W