Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture

Open-set face forgery detection poses significant security threats and presents substantial challenges for existing detection models. These detectors primarily have two limitations: they cannot generalize across unknown forgery domains and inefficiently adapt to new data. To address these issues, we introduce an approach that is both general and parameter-efficient for face forgery detection. It builds on the assumption that different forgery source domains exhibit distinct style statistics. Previous methods typically require fully fine-tuning pre-trained networks, consuming substantial time and computational resources. In turn, we design a forgery-style mixture formulation that augments the diversity of forgery source domains, enhancing the model’s generalizability across unseen domains. Drawing on recent advancements in vision transformers (ViT) for face forgery detection, we develop a parameter-efficient ViT-based detection model that includes lightweight forgery feature extraction modules and enables the model to extract global and local forgery clues simultaneously. We only optimize the inserted lightweight modules during training, maintaining the original ViT structure with its pre-trained ImageNet weights. This training strategy effectively preserves the informative pre-trained knowledge while flexibly adapting the model to the task of Deepfake detection. Extensive experimental results demonstrate that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters, representing an important step toward open-set Deepfake detection in the wild.

💡 Research Summary

This paper, titled “Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture,” addresses the critical challenges in real-world deepfake detection: poor generalization to unseen forgery methods (domains) and the computational inefficiency of adapting models to new data. The authors propose a novel, two-pronged approach that is both highly generalizable and parameter-efficient.

The core innovation lies in two designed components. First, the Forgery Style Mixture Module tackles the generalization issue. Based on the observation that domain gaps primarily affect the statistical “style” of fake faces rather than real ones, this module selectively mixes the feature statistics (mean and variance from normalization layers) among forgery images within a training batch. This augmentation strategy exposes the model to a more diverse range of forgery artifacts during training, fundamentally enhancing its ability to recognize novel manipulation techniques without compromising its understanding of real faces.

Second, the authors introduce a Forgery-aware Parameter-Efficient Fine-Tuning (PEFT) framework. Instead of fully fine-tuning a large pre-trained Vision Transformer (ViT), which is computationally expensive, they freeze the entire ViT backbone and inject lightweight, trainable modules. These include: 1) A CDC (Central Difference Convolution) Adapter inserted into the Feed-Forward Networks (FFNs), which is specifically designed to capture local forgery clues like blending boundaries and texture inconsistencies by computing pixel intensity differences. 2) LoRA (Low-Rank Adaptation) layers integrated into the self-attention modules, which efficiently adapt the model’s global contextual focus toward forgery-related patterns using low-rank matrix updates. This design allows the model to leverage the powerful general-purpose knowledge of the pre-trained ViT while specializing it for deepfake detection with only about 0.8% of the total parameters being trainable.

Extensive experiments validate the effectiveness of the proposed method, dubbed OSDFD. In cross-dataset evaluations on benchmarks like FaceForensics++, Celeb-DF, DFDC, and others, OSDFD achieves state-of-the-art generalization performance, often surpassing fully fine-tuned counterparts. Crucially, it delivers this superior performance with a drastically reduced number of trainable parameters, demonstrating an exceptional balance between efficiency and effectiveness (as visualized in the paper’s Figure 3). Additional experiments confirm the model’s robustness against common image degradations such as compression, blur, and noise. In conclusion, this work presents a significant step towards practical and scalable open-set deepfake detection by simultaneously advancing generalization capability through style mixture and adaptation efficiency through task-specific PEFT design.