Law and Adversarial Machine Learning

When machine learning systems fail because of adversarial manipulation, how should society expect the law to respond? Through scenarios grounded in adversarial ML literature, we explore how some aspects of computer crime, copyright, and tort law interface with perturbation, poisoning, model stealing and model inversion attacks to show how some attacks are more likely to result in liability than others. We end with a call for action to ML researchers to invest in transparent benchmarks of attacks and defenses; architect ML systems with forensics in mind and finally, think more about adversarial machine learning in the context of civil liberties. The paper is targeted towards ML researchers who have no legal background.

💡 Research Summary

The paper “Law and Adversarial Machine Learning” examines how existing legal frameworks intersect with various adversarial attacks on machine learning (ML) systems, and it offers concrete recommendations for ML researchers to help shape future policy. Structured around four attack categories—supply‑chain compromise, perturbation/poisoning, model stealing, and model inversion—the authors map each to relevant U.S. statutes and liability doctrines, illustrating where legal exposure is likely and where it remains ambiguous.

First, the authors analyze supply‑chain, perturbation, and poisoning attacks through the lens of the Computer Fraud and Abuse Act (CFAA). They argue that classic man‑in‑the‑middle tampering of pre‑trained models downloaded over insecure HTTP, buffer‑overflow exploits in OpenCV that cause misclassification, and health‑care data poisoning that alters patient dosage recommendations can all be framed as “unauthorized access” and “damage” under the CFAA. However, the statute’s definition of “damage” is unsettled when the harm is a degraded model performance rather than physical destruction, creating uncertainty about prosecutorial discretion.

Second, the paper turns to intellectual‑property law for model stealing and inversion attacks. Using the Fredrikson et al. reconstruction of private training data as a case study, the authors note that factual data (e.g., medical records) are not copyrightable in the United States, so a plaintiff would struggle to succeed on a copyright claim. By contrast, copyrighted media (images, audio) that are reconstructed could give rise to infringement claims. Regarding the model itself, software copyright protects the specific expression of code, not the underlying function; therefore a reconstructed model that differs in implementation is unlikely to infringe. Trade‑secret law may offer protection if the owner can demonstrate reasonable secrecy measures, but enforcement is limited when the model is accessed via a public API and later redistributed by a third party. Contractual terms of service can supplement protection against API users but do not shield against public releases of stolen models.

Third, the authors explore product‑liability and negligence doctrines in the context of adversarial examples. They cite the European Commission’s upcoming AI liability framework (mid‑2019) as an indicator that regulatory pressure is building. Using a consumer‑grade drone image‑recognition system as an example, they discuss whether a manufacturer could be held negligent for deploying a model with non‑zero test error that is known to be vulnerable to adversarial inputs. Existing case law on software liability is sparse, and the lack of industry‑wide security standards makes it difficult for courts to assess what constitutes “reasonable” safeguards. Moreover, the complex supply chain—mixing open‑source libraries, commercial cloud services, and third‑party models—complicates attribution of fault when a failure occurs.

Finally, the paper offers three actionable recommendations for ML researchers: (1) develop transparent, standardized benchmarks of attacks and defenses, and adopt risk‑scoring frameworks (e.g., DREAD) to help policymakers understand the technical landscape; (2) design ML systems with forensic capabilities—robust logging, intrusion alerts, sandboxed response environments—to facilitate attribution and legal evidence collection; (3) consider civil‑liberties implications, recognizing that adversarial techniques can both empower dissidents (e.g., 3‑D‑printed glasses to evade facial recognition) and enable authoritarian backdoors. By aligning research practices with these legal insights, the community can reduce uncertainty, promote responsible deployment, and influence the evolution of law in a way that safeguards both innovation and societal values.