Closing the Confusion Loop: CLIP-Guided Alignment for Source-Free Domain Adaptation

Source-Free Domain Adaptation (SFDA) tackles the problem of adapting a pre-trained source model to an unlabeled target domain without accessing any source data, which is quite suitable for the field of data security. Although recent advances have shown that pseudo-labeling strategies can be effective, they often fail in fine-grained scenarios due to subtle inter-class similarities. A critical but underexplored issue is the presence of asymmetric and dynamic class confusion, where visually similar classes are unequally and inconsistently misclassified by the source model. Existing methods typically ignore such confusion patterns, leading to noisy pseudo-labels and poor target discrimination. To address this, we propose CLIP-Guided Alignment(CGA), a novel framework that explicitly models and mitigates class confusion in SFDA. Generally, our method consists of three parts: (1) MCA: detects first directional confusion pairs by analyzing the predictions of the source model in the target domain; (2) MCC: leverages CLIP to construct confusion-aware textual prompts (e.g. a truck that looks like a bus), enabling more context-sensitive pseudo-labeling; and (3) FAM: builds confusion-guided feature banks for both CLIP and the source model and aligns them using contrastive learning to reduce ambiguity in the representation space. Extensive experiments on various datasets demonstrate that CGA consistently outperforms state-of-the-art SFDA methods, with especially notable gains in confusion-prone and fine-grained scenarios. Our results highlight the importance of explicitly modeling inter-class confusion for effective source-free adaptation. Our code can be find at https://github.com/soloiro/CGA

💡 Research Summary

The paper tackles Source‑Free Domain Adaptation (SFDA), a setting where only a pretrained source classifier is available and no source‑domain data can be accessed, a scenario increasingly relevant for privacy‑sensitive applications. Existing SFDA approaches rely heavily on pseudo‑labeling based on the source model’s soft predictions, assuming that target samples form well‑separated clusters aligned with semantic classes. This assumption breaks down in fine‑grained tasks where inter‑class visual differences are subtle, leading to noisy pseudo‑labels and poor target performance. Moreover, the authors identify a largely overlooked phenomenon: asymmetric and dynamic class confusion. Certain classes are more prone to being mis‑classified into specific other classes (e.g., “truck → bus” but not the reverse), and these confusion patterns evolve during training. Prior methods treat inter‑class relations as symmetric and static, thus ignoring this nuance.

To address these issues, the authors propose CLIP‑Guided Alignment (CGA), a three‑stage framework that explicitly detects, represents, and resolves class confusion without any source data.

1. Model Class Confusion Analysis (MCA) – By feeding the entire unlabeled target set through the source model, CGA collects soft prediction vectors and computes class‑conditional probability centroids. From these centroids it builds a directed confusion matrix (or graph) whose edges quantify how often class i is predicted as class j. This matrix is updated epoch‑wise, capturing the dynamic, asymmetric nature of confusion.

2. Multi‑Prototype Confused CLIP (MCC) – Leveraging the vision‑language model CLIP, CGA constructs confusion‑aware textual prompts for each high‑weight confusion pair, such as “a truck that looks like a bus”. These prompts are encoded by CLIP’s text encoder to obtain hybrid semantic prototypes that embed the ambiguity between the two classes. Only a small set of prompt parameters is fine‑tuned (following CoOp), while the bulk of CLIP remains frozen. The hybrid prototypes are then used to generate refined pseudo‑labels for target images, effectively correcting the source model’s biased predictions.

3. Feature Alignment Module (FAM) – Using the confusion graph, CGA builds two confusion‑aware feature banks: one from the source model’s feature extractor and one from CLIP’s image encoder. Each bank stores class‑center vectors for the confusion‑aware pseudo‑classes. A contrastive loss aligns the two banks, pulling together corresponding class centers and pushing apart different ones. This alignment transfers CLIP’s rich semantic priors into the source model’s feature space, reducing inter‑class overlap and producing a more discriminative target representation.

The three stages form a closed loop: confusion perception informs prompt generation, prompts improve pseudo‑labels, and improved pseudo‑labels lead to better feature alignment, which in turn refines the confusion perception in the next epoch.

Experimental validation is performed on four widely used SFDA benchmarks—Office‑Home, VisDA‑2017, DomainNet, and the fine‑grained CUB‑200‑2011 dataset. CGA consistently outperforms state‑of‑the‑art SFDA methods (e.g., CoWA‑JMDS, PLUE, CRS) across all settings, with especially large gains (3–5 percentage points) on datasets where class confusion is pronounced. Ablation studies confirm that each component (MCA, MCC, FAM) contributes positively, and visualizations of the evolving confusion matrix and t‑SNE embeddings illustrate how CGA quickly suppresses dominant confusion directions and yields well‑separated class clusters.

Beyond performance, CGA respects privacy constraints: no source images are required, and CLIP is used as a publicly available pretrained model, avoiding any leakage of proprietary data.

In summary, the paper makes four key contributions: (1) it formally defines and quantifies asymmetric, dynamic class confusion in SFDA; (2) it introduces a novel CLIP‑guided prompting mechanism that encodes this confusion into textual prototypes; (3) it proposes a contrastive feature‑space alignment that bridges the source model and CLIP, mitigating ambiguity; and (4) it demonstrates that explicitly modeling confusion yields state‑of‑the‑art results on both generic and fine‑grained domain adaptation tasks. This work opens a promising direction for future SFDA research, emphasizing the importance of understanding and correcting class‑specific misclassifications rather than treating all classes uniformly.