Evaluating False Alarm and Missing Attacks in CAN IDS

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.

💡 Research Summary

This paper presents a systematic adversarial robustness evaluation of machine‑learning‑based intrusion detection systems (IDS) for the automotive Controller Area Network (CAN). Using the high‑fidelity ROAD dataset, which contains over 1.5 million benign frames and nearly 50 k malicious frames spanning a variety of realistic attacks (fuzzing floods, engine‑coolant temperature spoofing, speedometer manipulation, reverse‑light toggling, correlated‑signal attacks, etc.), the authors train five classifiers: four shallow models (Decision Tree, Random Forest, Extra Trees, XGBoost) and a deep neural network (DNN) composed of four fully‑connected layers (16 neurons each) with a sigmoid output.

The study adopts a realistic threat model: an attacker with white‑box knowledge of the IDS can inject or modify CAN payload bytes while preserving the identifier (ID) and data‑length code (DLC) to remain protocol‑compliant. Adversarial perturbations are generated exclusively on the eight data bytes (0‑255 range) and clipped to valid values, ensuring that every crafted frame could actually appear on the bus. Three classic gradient‑based attacks are employed via the Adversarial Robustness Toolbox: Fast Gradient Sign Method (FGSM, single‑step), Basic Iterative Method (BIM, multi‑step), and Projected Gradient Descent (PGD, multi‑step with random restarts).

Baseline results on clean data show high accuracy (>95 %) and low false‑positive/false‑negative rates for all models, with the DNN achieving the best overall detection performance. However, when adversarial perturbations are applied, two critical failure modes emerge. First, “false‑alarm induction” occurs when benign frames are subtly altered so that the IDS flags them as malicious, potentially leading to unnecessary mitigation actions and erosion of driver trust. Second, “missed‑attack induction” (false negatives) happens when malicious frames are perturbed just enough to slip past the detector, directly endangering vehicle safety.

Experimental findings reveal that all models are relatively robust to false‑alarm induction but are vulnerable to missed‑attack induction. The DNN, while excelling on clean traffic, suffers the largest increase in false‑negative rate under FGSM, BIM, and PGD attacks, especially as the perturbation magnitude (ε) grows beyond 0.1. In contrast, the Extra Trees (ET) ensemble exhibits the smallest rise in missed‑attack rate across multi‑step attacks, suggesting that certain shallow ensemble architectures can provide superior resilience to gradient‑based evasion. Random Forest and XGBoost display intermediate behavior, whereas the simple Decision Tree is the most fragile.

The authors also analyze the impact of attack strength: as ε increases, the false‑negative rate for all models escalates sharply, with some configurations exceeding a 30 % miss rate, underscoring that even modest payload manipulations can critically undermine IDS reliability. By jointly evaluating false positives and false negatives, the paper highlights a gap in prior work that typically examined only one failure mode.

Key contributions include: (1) a protocol‑aware adversarial generation method that respects CAN field constraints; (2) a unified evaluation framework that simultaneously measures false‑alarm and missed‑attack induction; (3) a comprehensive comparison of shallow versus deep learning IDS under realistic attack scenarios; (4) evidence that extra‑trees ensembles may offer better adversarial robustness than deep networks in this domain; and (5) open‑source release of the implementation and evaluation pipeline to foster reproducibility.

The study concludes that current CAN‑IDS solutions are not sufficiently hardened against adversarial manipulation, particularly regarding evasion attacks that cause missed detections. Future work should explore temporal feature extraction, adversarial training, ensemble diversification, and defense‑in‑depth strategies to mitigate both false alarms and missed attacks, thereby enhancing the safety and trustworthiness of in‑vehicle networks.