From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Online Test-Time Backdoor Defense

Deep Neural Networks remain inherently vulnerable to backdoor attacks. Traditional test-time defenses largely operate under the paradigm of internal diagnosis methods like model repairing or input robustness, yet these approaches are often fragile under advanced attacks as they remain entangled with the victim model’s corrupted parameters. We propose a paradigm shift from Internal Diagnosis to External Semantic Auditing, arguing that effective defense requires decoupling safety from the victim model via an independent, semantically grounded auditor. To this end, we present a framework harnessing Universal Vision-Language Models (VLMs) as evolving semantic gatekeepers. We introduce PRISM (Prototype Refinement & Inspection via Statistical Monitoring), which overcomes the domain gap of general VLMs through two key mechanisms: a Hybrid VLM Teacher that dynamically refines visual prototypes online, and an Adaptive Router powered by statistical margin monitoring to calibrate gating thresholds in real-time. Extensive evaluation across 17 datasets and 11 attack types demonstrates that PRISM achieves state-of-the-art performance, suppressing Attack Success Rate to <1% on CIFAR-10 while improving clean accuracy, establishing a new standard for model-agnostic, externalized security.

💡 Research Summary

The paper tackles the persistent vulnerability of deep neural networks to backdoor attacks, highlighting the shortcomings of existing test‑time defenses that rely on internal diagnosis—either model‑repair techniques that inspect internal activations or input‑robustness methods that assume triggers are fragile to perturbations. Modern attacks (dynamic triggers, clean‑label, clean‑image, and adaptive flooding) deliberately hide their footprints, rendering these internal‑centric approaches ineffective. To break this dependency, the authors propose a paradigm shift: External Semantic Auditing. Instead of asking a potentially compromised model to validate its own predictions, an independent, frozen universal vision‑language model (VLM) serves as a clean auditor, leveraging its open‑world semantic knowledge that is statistically independent of the victim model’s poisoned distribution.

The core contribution is PRISM (Prototype Refinement & Inspection via Statistical Monitoring), the first test‑time framework that wraps a suspicious classifier with a dual‑stream architecture. One stream processes inputs through the victim model, the other through a “Hybrid VLM Teacher.” The teacher augments static text anchors with visual prototypes that are learned online from the test stream, thereby bridging the domain gap between generic VLM zero‑shot performance and the specialized task at hand. Prototypes (class centroids) are updated via a cumulative moving average, requiring no labeled data. The two logit sets (text‑based and prototype‑based) are fused with a tunable weight λ, yielding robust VLM logits.

An Adaptive Router then decides, for each sample, whether to trust the victim model’s prediction or to replace it with the VLM teacher’s output. The router monitors the logit margin distribution in real time, modeling it with a Cornish‑Fisher expansion to capture mean, variance, skewness, and kurtosis. Based on the current statistical quantiles, a dynamic threshold is set; if the margin falls outside a safe region, the sample is deemed suspicious and the VLM prediction is used. This online statistical monitoring eliminates the need for static thresholds, which are brittle across heterogeneous data and attack modalities.

The method operates entirely at inference time, without access to training data, model weights, or any fine‑tuning of the VLM (the VLM remains frozen). Experiments span 17 datasets (CIFAR‑10/100, ImageNet‑C, GTSRB, SVHN, etc.) and 11 attack types (BadNet, WaNet, Dynamic Trigger, Clean‑Label, Clean‑Image, Adaptive Flooding, etc.). Six VLM backbones are evaluated, including CLIP, SigLIP, Qwen‑VL, and Gemma‑3. PRISM consistently suppresses Attack Success Rate (ASR) to below 1 % while improving clean accuracy (CA) by an average of 0.3 % points. Notably, against clean‑image and adaptive flooding attacks—where prior model‑repair and input‑purification methods fail dramatically—PRISM maintains high detection rates. Computational overhead is modest: a KV‑Cache for generative VLMs and cumulative moving averages keep extra latency to roughly 5–7 ms per sample on a modern GPU.

Limitations are acknowledged. The VLM’s pre‑training data may introduce cultural or domain bias, potentially affecting sensitivity to niche triggers. The statistical router requires a sufficient number of test samples to estimate margin distributions reliably; abrupt distribution shifts can cause temporary threshold lag. The current work focuses on image classification; extending to detection, segmentation, or multimodal tasks will require additional engineering.

Future directions include integrating multimodal auditors (text, audio, video), employing meta‑learning to predict optimal thresholds, and designing Bayesian online estimators for faster adaptation. The authors also suggest lightweight domain adapters or prompt‑tuning to mitigate VLM bias.

In summary, PRISM demonstrates that decoupling defense from the compromised model via an external VLM auditor, combined with online prototype refinement and statistical margin monitoring, yields a model‑agnostic, data‑free, and robust test‑time backdoor defense. This approach sets a new benchmark for securing deployed AI services against sophisticated backdoor threats.