Comparative Insights on Adversarial Machine Learning from Industry and Academia: A User-Study Approach

An exponential growth of Machine Learning and its Generative AI applications brings with it significant security challenges, often referred to as Adversarial Machine Learning (AML). In this paper, we conducted two comprehensive studies to explore the perspectives of industry professionals and students on different AML vulnerabilities and their educational strategies. In our first study, we conducted an online survey with professionals revealing a notable correlation between cybersecurity education and concern for AML threats. For our second study, we developed two CTF challenges that implement Natural Language Processing and Generative AI concepts and demonstrate a poisoning attack on the training data set. The effectiveness of these challenges was evaluated by surveying undergraduate and graduate students at Carnegie Mellon University, finding that a CTF-based approach effectively engages interest in AML threats. Based on the responses of the participants in our research, we provide detailed recommendations emphasizing the critical need for integrated security education within the ML curriculum.

💡 Research Summary

This paper investigates the gap between industry practitioners and academia regarding Adversarial Machine Learning (AML) threats and proposes an educational approach to bridge it. The authors conduct two complementary user‑studies.

Study 1 is an online survey of twelve professionals working in cybersecurity, machine learning, or privacy. Participants (predominantly aged 25‑34) provided demographic information, their background in security, ML, and privacy, and their level of concern about AML threats. Five hypotheses (H1‑H5) examine whether education, experience, non‑technical factors (time, ethics, cost), and technical factors (accuracy, performance) influence AML awareness and adoption of defensive practices. The survey, approved by an Institutional Review Board and compensated at $5 per ten‑minute response, reveals a positive correlation between formal security/ML education and heightened concern for AML. Respondents also report occasional use of Capture‑the‑Flag (CTF) platforms for self‑education, but the small sample size limits statistical generalization.

Study 2 develops two CTF challenges hosted on the picoCTF platform that illustrate training‑phase poisoning attacks on natural‑language and generative‑AI models. Challenge 1 implements label‑flipping and clean‑label poisoning; Challenge 2 uses Feature Collision and Convex Polytope attacks to manipulate a chatbot’s training loop. The challenges were deployed to 45 undergraduate and graduate students at Carnegie Mellon University. Pre‑ and post‑challenge surveys, together with success‑rate metrics, show that participants found the CTF format engaging and that their self‑reported understanding of AML threats increased significantly after the exercise. The authors argue that hands‑on, competition‑style learning outperforms purely theoretical instruction for AML concepts. However, the study lacks rigorous control of prior knowledge, does not publish detailed success statistics, and its findings may not generalize beyond a single institution.

The related‑work section surveys AML taxonomies (influence, security violation, specificity), attack phases (training vs. testing), and specific poisoning techniques (label flipping, clean‑label, backdoor, Feature Collision, Convex Polytope). Defensive strategies such as adversarial training, detection mechanisms, robust optimization (e.g., Projected Gradient Descent), regularization, and feature squeezing are summarized, emphasizing that a layered defense yields the strongest empirical robustness. The authors also cite educational research showing the need for AI‑security curricula at K‑12 and higher‑education levels.

Key contributions are: (1) a comparative analysis of industry and student perceptions of AML, (2) the design and empirical evaluation of AML‑focused CTF challenges, and (3) concrete recommendations for integrating AML security education into machine‑learning curricula. Recommendations include making security and privacy modules mandatory in ML courses, incorporating CTF‑style labs into regular coursework, establishing organizational AML policies that address both technical and non‑technical constraints, and encouraging continuous professional development through competitive platforms.

Limitations acknowledged by the authors involve the modest sample size and potential selection bias in Study 1, the single‑site nature of Study 2, the primarily descriptive statistical analysis without hypothesis‑testing significance values, and the absence of recent large‑scale language‑model defense techniques. Future work is suggested to involve larger, more diverse industry samples, multi‑institutional CTF deployments, and rigorous quantitative evaluation (e.g., ANOVA, confidence intervals) to validate the educational impact.

In summary, the paper provides empirical evidence that industry professionals with security‑focused education are more aware of AML threats, and that CTF‑based, hands‑on learning significantly improves student engagement and understanding of AML. By combining these insights, the authors propose a roadmap for embedding AML defense strategies into ML education, aiming to produce a workforce better equipped to secure emerging AI systems.