Trojan Cleansing with Neural Collapse

Trojan attacks are sophisticated training-time attacks on neural networks that embed backdoor triggers which force the network to produce a specific output on any input which includes the trigger. With the increasing relevance of deep networks which are too large to train with personal resources and which are trained on data too large to thoroughly audit, these training-time attacks pose a significant risk. In this work, we connect trojan attacks to Neural Collapse, a phenomenon wherein the final feature representations of over-parameterized neural networks converge to a simple geometric structure. We provide experimental evidence that trojan attacks disrupt this convergence for a variety of datasets and architectures. We then use this disruption to design a lightweight, broadly generalizable mechanism for cleansing trojan attacks from a wide variety of different network architectures and experimentally demonstrate its efficacy.

💡 Research Summary

This paper presents a novel and effective method for defending against Trojan (backdoor) attacks in deep neural networks by leveraging the recently discovered phenomenon of Neural Collapse (NC). The core premise is establishing a first-of-its-kind connection between the asymmetric nature of Trojan attacks and the symmetric geometric structure that emerges during NC.

The authors begin by detailing the threat model: an adversary poisons a small fraction of the training data by embedding a trigger and relabeling those samples to a target class. This causes the trained model to behave normally on clean inputs but to misclassify any triggered input to the target class. The defense challenge is to “cleanse” a potentially Trojaned model without prior knowledge of the trigger pattern, architecture details, or full training data, while preserving clean-data accuracy.

The key insight stems from Neural Collapse, a phenomenon observed in over-parameterized networks trained beyond zero training error. In this “terminal phase of training,” the last-layer features of each class collapse to their means, and both these class means and the classifier weights converge to a Simplex Equiangular Tight Frame (ETF)—a highly symmetric and maximally separated geometric configuration. The paper hypothesizes and provides extensive experimental evidence that the injection of a Trojan disrupts this natural convergence towards symmetry. Across multiple datasets (CIFAR-10/100, GTSRB) and architectures, Trojaned models show consistently weaker NC, quantified by standard metrics (NC1-NC4). Notably, the weight vector and feature mean corresponding to the attack target class often exhibit a smaller norm, breaking the equinorm property of a perfect ETF.

Leveraging this disruption, the authors propose a lightweight cleansing algorithm. Instead of relying on model compression (which may harm performance) or trigger reconstruction (which requires assumptions about the trigger), their method directly enforces the symmetric properties of NC on the potentially corrupted model. The core operation is a targeted adjustment of the final classification layer’s weight matrix. The algorithm optimizes these weights to better align with the structure of an ETF (improving NC2 metrics) and to enhance their duality with the feature means (improving NC3). This process effectively “repairs” the asymmetry introduced by the Trojan, neutralizing the backdoor pathway without altering the model’s core feature extractor.

The experimental evaluation demonstrates state-of-the-art or competitive performance. The method effectively mitigates various attack types (patch-based, filter-based) on different network architectures, including ResNets and Vision Transformers. It maintains high clean-data accuracy while drastically reducing the attack success rate. Notably, it shows superior robustness against sophisticated attacks like WaNet and Refool, and performs well under conditions of data imbalance. The algorithm’s strength lies in its simplicity, generality, and low computational cost, making it a practical defense for scenarios where users have limited ML expertise—precisely the setting most vulnerable to Trojan attacks.

In summary, this work makes a significant contribution by bridging theoretical understanding of neural network training dynamics (Neural Collapse) with the pressing practical problem of AI security. It offers a principled, efficient, and broadly applicable defense mechanism against backdoor attacks.