Beyond Crash: Hijacking Your Autonomous Vehicle for Fun and Profit

Autonomous Vehicles (AVs), especially vision-based AVs, are rapidly being deployed without human operators. As AVs operate in safety-critical environments, understanding their robustness in an adversarial environment is an important research problem. Prior physical adversarial attacks on vision-based autonomous vehicles predominantly target immediate safety failures (e.g., a crash, a traffic-rule violation, or a transient lane departure) by inducing a short-lived perception or control error. This paper shows a qualitatively different risk: a long-horizon route integrity compromise, where an attacker gradually steers a victim AV away from its intended route and into an attacker-chosen destination while the victim continues to drive “normally.” This will not pose a danger to the victim vehicle itself, but also to potential passengers sitting inside the vehicle. In this paper, we design and implement the first adversarial framework, called JackZebra, that performs route-level hijacking of a vision-based end-to-end driving stack using a physically plausible attacker vehicle with a reconfigurable display mounted on the rear. The central challenge is temporal persistence: adversarial influence must remain effective in changing viewpoints, lighting, weather, traffic, and the victim’s continual replanning – without triggering conspicuous failures. Our key insight is to treat route hijacking as a closed-loop control problem and to convert adversarial patches into steering primitives that can be selected online via an interactive adjustment loop. Our adversarial patches are also carefully designed against worst-case background and sensor variations so that the adversarial impacts on the victim. Our evaluation shows that JackZebra can successfully hijack victim vehicles to deviate from original routes and stop at adversarial destinations with a high success rate.

💡 Research Summary

The paper introduces a novel physical adversarial attack on vision‑based end‑to‑end autonomous driving systems, targeting the long‑term integrity of a vehicle’s route rather than causing an immediate safety failure. The authors propose “JackZebra,” a framework in which an attacker‑controlled vehicle equipped with a reconfigurable rear‑mounted display projects carefully crafted adversarial patches toward the victim vehicle’s front‑facing camera. By treating each patch as a “steering primitive,” the system can continuously bias the victim’s perception‑planning‑control loop, gradually steering it away from its intended destination toward a location chosen by the attacker.

JackZebra operates in two stages. In the offline stage, a “patch bank” is generated using map data and street‑view imagery of the target town. Each patch is optimized via a min‑max formulation that accounts for worst‑case background, lighting, weather, and sensor variations, ensuring robustness across diverse environmental conditions. The patches are designed to induce specific turning angles (e.g., small left‑turn bias, larger right‑turn bias) when observed by the victim’s camera.

During the online attack, the adversarial vehicle follows the victim at a safe distance, continuously monitoring the victim’s behavior with a rear‑facing camera and GPS. A closed‑loop adjustment algorithm compares the observed trajectory against a prediction model; if the victim deviates less (or more) than desired, the system selects a different patch from the bank and displays it on the rear screen. Simultaneously, a front‑facing camera and LiDAR keep the attacker vehicle compliant with traffic rules (stopping at red lights, yielding at signs) so that the overall maneuver remains stealthy.

The core technical challenge is temporal persistence: the adversarial influence must survive many perception‑planning‑control cycles despite changing viewpoints, distances, illumination, weather, and surrounding traffic. By encoding patches as controllable steering primitives rather than static misclassifications, JackZebra can predict the magnitude of bias each patch will produce and adjust in real time, achieving a coherent long‑horizon manipulation.

Evaluation is performed in a high‑fidelity simulation environment that integrates the CARLA simulator, SimLingo driving agent, and the Bench2Drive benchmark. The authors test 39 original‑adversarial route pairs; JackZebra successfully hijacks 34 of them (≈87 % success) and completes an average hijacked distance of 122.2 m. The hijacked trajectories obey traffic regulations, exhibit smooth acceleration profiles, and do not contain conspicuous anomalies, making detection by passengers or external observers difficult.

The paper’s contributions are fourfold: (1) definition of a long‑term route‑hijacking threat model, (2) a robust min‑max patch generation method that tolerates worst‑case environmental variations, (3) an interactive, feedback‑driven patch‑selection loop that operates in real time, and (4) a comprehensive simulation‑based validation demonstrating high success rates and stealth. The work highlights a critical vulnerability of camera‑centric autonomous driving stacks: an attacker can exploit benign‑looking visual content to exert subtle yet persistent control over vehicle navigation.

Future defense directions suggested include strengthening multi‑sensor fusion (e.g., incorporating LiDAR, radar, and GPS to cross‑validate visual cues), developing real‑time adversarial‑patch detectors, and leveraging vehicle‑to‑vehicle communication for collaborative anomaly detection. Overall, JackZebra expands the adversarial attack landscape from momentary perception errors to sustained, covert route manipulation, urging the autonomous driving community to reconsider security assumptions around visual perception pipelines.