Fine-Grained Frame Modeling in Multi-head Self-Attention for Speech Deepfake Detection

Transformer-based models have shown strong performance in speech deepfake detection, largely due to the effectiveness of the multi-head self-attention (MHSA) mechanism. MHSA provides frame-level attention scores, which are particularly valuable because deepfake artifacts often occur in small, localized regions along the temporal dimension of speech. This makes fine-grained frame modeling essential for accurately detecting subtle spoofing cues. In this work, we propose fine-grained frame modeling (FGFM) for MHSA-based speech deepfake detection, where the most informative frames are first selected through a multi-head voting (MHV) module. These selected frames are then refined via a cross-layer refinement (CLR) module to enhance the model’s ability to learn subtle spoofing cues. Experimental results demonstrate that our method outperforms the baseline model and achieves Equal Error Rate (EER) of 0.90%, 1.88%, and 6.64% on the LA21, DF21, and ITW datasets, respectively. These consistent improvements across multiple benchmarks highlight the effectiveness of our fine-grained modeling for robust speech deepfake detection.

💡 Research Summary

The paper addresses a critical limitation of current transformer‑based speech deepfake detection systems: while multi‑head self‑attention (MHSA) provides frame‑level attention scores, the standard practice aggregates these scores globally, which can dilute short, localized artifacts that are characteristic of synthetic speech. To overcome this, the authors propose a Fine‑Grained Frame Modeling (FGFM) framework that explicitly selects and refines the most informative frames before they are used for final classification.

The baseline model is an XLSR‑Conformer architecture that combines a pre‑trained XLS‑R feature extractor with a Conformer encoder. Each Conformer block contains an MHSA module with K attention heads. The first component of FGFM, the Multi‑Head Voting (MHV) module, treats each head as a weak learner in a bagging‑style ensemble. For every block, the attention map between the classification token and all T frames is obtained for each head (A₁…A_K). From each head, the top v frames with the highest attention scores are voted as “important” and marked with a binary mask M_k. The masks from all heads are summed to produce M′, which is then smoothed by a 1‑D Gaussian‑like convolution kernel G, yielding a refined score map M*. The v frames with the highest values in M* are extracted as the selected frame representations X_sel^l. Empirically, v = 24 provides the best trade‑off for 4‑second audio segments.

The second component, Cross‑Layer Refinement (CLR), aggregates information across layers. At the (L + 1)‑th Conformer block, the input consists of the current classification token concatenated with all previously selected frames from blocks 1…L. The block’s output, denoted f_cross, is again processed by the MHV module to obtain a second set of selected frames. These are fed into an additional (L + 2)‑th block together with the classification token, producing a refined feature f_refined. Two parallel cross‑attention operations exchange information between f_cross and f_refined, after which a lightweight Dynamic Aggregation Feed‑Forward (D‑AFF) block merges the two streams. The resulting enriched classification token is finally passed to a binary classification head.

Experiments were conducted on three benchmarks: ASVspoof 2021 LA (LA21), ASVspoof 2021 DF (DF21), and an out‑of‑domain In‑the‑Wild (ITW) dataset. All models were trained on the ASVspoof 2019 LA training set and evaluated using Equal Error Rate (EER). The baseline XLSR‑Conformer achieved 0.97 % (LA21), 2.58 % (DF21), and 8.42 % (ITW). Incorporating FGFM reduced these to 0.90 %, 1.88 %, and 6.64 % respectively, corresponding to relative improvements of 7.2 %, 27.1 %, and 21.1 %. The method also outperformed several recent state‑of‑the‑art systems, including XLSR‑Mamba (0.93 % LA, 1.88 % DF, 6.71 % ITW) and other SSL‑based models. Notably, FGFM maintained its advantage when the Conformer encoder was swapped for a standard Transformer, demonstrating architectural agnosticism.

Ablation studies confirmed the contribution of each component. Removing the D‑AFF module increased EER by 5–8 % across datasets, while omitting the Gaussian‑kernel refinement in MHV caused a 5–15 % degradation, highlighting the importance of both cross‑layer integration and attention‑score smoothing. Varying the number of voted frames per head (v = 16, 24, 32, 40) showed that performance peaked at v = 24; larger values introduced noisy or silent frames, reducing discriminative power. Visualizations of selected frames overlaid on spectrograms revealed that MHV consistently focuses on high‑energy speech regions while avoiding silence, aligning with known locations of spoofing artifacts.

In summary, the paper introduces a novel, modular approach to enhance frame‑level granularity in MHSA‑based speech deepfake detectors. By combining multi‑head voting with cross‑layer refinement and a dynamic aggregation block, the system captures subtle, temporally localized cues that are otherwise lost in global attention pooling. The extensive experiments across in‑domain and out‑of‑domain benchmarks validate the effectiveness and generality of the approach. Future directions suggested include adaptive selection of v, dynamic kernel shaping, real‑time deployment considerations, and integration with other self‑supervised speech backbones such as HuBERT or Data2Vec.