FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Time

This paper proposes FreeFuse, a training-free framework for multi-subject text-to-image generation through automatic fusion of multiple subject LoRAs. In contrast to prior studies that focus on retraining LoRA to alleviate feature conflicts, our analysis reveals that simply spatially confining the subject LoRA’s output to its target region and preventing other LoRAs from directly intruding into this area is sufficient for effective mitigation. Accordingly, we implement Adaptive Token-Level Routing during the inference phase. We introduce FreeFuseAttn, a mechanism that exploits the flow matching model’s intrinsic semantic alignment to dynamically match subject-specific tokens to their corresponding spatial regions at early denoising timesteps, thereby bypassing the need for external segmentors. FreeFuse distinguishes itself through high practicality: it necessitates no additional training, model modifications, or user-defined masks spatial conditions. Users need only provide subject activation words to achieve seamless integration into standard workflows. Extensive experiments validate that FreeFuse outperforms existing approaches in both identity preservation and compositional fidelity. Our code is available at https://github.com/yaoliliu/FreeFuse.

💡 Research Summary

FreeFuse introduces a training‑free framework for multi‑subject text‑to‑image generation that resolves the long‑standing problem of feature conflicts when multiple LoRA adapters are combined. The authors first observe that the root cause of conflicts is the indiscriminate broadcasting of LoRA parameter updates (Δθ) to every token in the latent space. By simply masking each LoRA’s contribution to the spatial region that corresponds to its subject, they can prevent overlapping updates. This insight is formalized in Equation (1), where an indicator function I(p ∈ R_i) ensures that only the LoRA associated with region R_i modifies token p.

A potential objection is that self‑attention could still propagate features across regions. The paper leverages the well‑known locality property of DiT‑based diffusion/flow‑matching models: early layers aggregate global context, but deeper semantic layers exhibit strong diagonal dominance, meaning tokens attend primarily to other tokens within the same spatial region. Empirical analysis (Figure 4) shows intra‑subject attention becoming 7‑times larger than inter‑subject attention in later blocks, confirming that cross‑region contamination is negligible when masks are applied.

FreeFuseAttn exploits this intrinsic segmentation capability. By focusing on the early‑to‑mid denoising timesteps—identified as the window where the model’s cross‑modal alignment is strongest—the method extracts token‑level masks from a combination of cross‑attention maps and token similarity matrices. Unlike prior CrossAttn, ConceptAttn, or SP‑Attn approaches, FreeFuseAttn fuses semantic‑driven attention with cohesion‑driven similarity, eliminating “hole” artifacts and producing contiguous masks. Quantitative comparison against SAM‑generated ground‑truth masks shows superior Precision@K scores.

The second phase introduces a token‑level router and an attention bias. The router enforces a strict exclusivity constraint: each spatial token is modulated by at most one subject‑specific LoRA, as dictated by the masks from Phase 1. Simultaneously, an attention bias derived from the same masks nudges the model’s cross‑attention to align more tightly with the intended subject semantics, dramatically reducing concept bleeding. This two‑step pipeline—mask extraction → router + bias—operates entirely at inference time, requiring no extra parameters, no model architecture changes, and no external segmentation networks.

FreeFuse is designed to be plug‑and‑play with existing control modules such as ControlNet, IP‑Adapter, and Style LoRAs. Users only need to supply activation words for each subject (e.g., “a photo of ”) and the system automatically handles spatial routing. Experiments demonstrate that even with six concurrent LoRAs, FreeFuse produces clean compositions without the artifacts, distortion, or identity loss observed in prior methods (e.g., ZipLoRA, K‑LoRA, CLoRA, LoRAShop). Both identity preservation metrics and compositional fidelity scores surpass state‑of‑the‑art baselines, and human evaluations confirm more natural layouts and faithful subject rendering.

In summary, FreeFuse contributes three key innovations: (1) a theoretical and empirical justification that spatial masking of LoRA updates suffices to mitigate multi‑subject conflicts; (2) FreeFuseAttn, a robust attention‑based intrinsic segmentation technique that yields high‑quality token masks without external models; (3) a token‑level routing and bias mechanism that enforces exclusive LoRA activation per token, eliminating feature interference while preserving global coherence. This training‑free, inference‑only solution dramatically lowers the barrier for multi‑subject personalized generation, offering a practical and scalable alternative to retraining‑heavy or mask‑heavy approaches.