Chasing Shadows: Pitfalls in LLM Security Research

Large language models (LLMs) are increasingly prevalent in security research. Their unique characteristics, however, introduce challenges that undermine established paradigms of reproducibility, rigor, and evaluation. Prior work has identified common pitfalls in traditional machine learning research, but these studies predate the advent of LLMs. In this paper, we identify nine common pitfalls that have become (more) relevant with the emergence of LLMs and that can compromise the validity of research involving them. These pitfalls span the entire computation process, from data collection, pre-training, and fine-tuning to prompting and evaluation. We assess the prevalence of these pitfalls across all 72 peer-reviewed papers published at leading Security and Software Engineering venues between 2023 and 2024. We find that every paper contains at least one pitfall, and each pitfall appears in multiple papers. Yet only 15.7% of the present pitfalls were explicitly discussed, suggesting that the majority remain unrecognized. To understand their practical impact, we conduct four empirical case studies showing how individual pitfalls can mislead evaluation, inflate performance, or impair reproducibility. Based on our findings, we offer actionable guidelines to support the community in future work.

💡 Research Summary

The paper “Chasing Shadows: Pitfalls in LLM Security Research” identifies nine distinct pitfalls that arise specifically when large language models (LLMs) are used in security‑related research. While prior work has catalogued generic machine‑learning failures, the scale, opacity, and prompt‑driven interaction of modern LLMs create new failure modes that can jeopardize reproducibility, rigor, and the validity of reported results.

The authors first map the typical LLM development pipeline into five stages: (1) data collection and labeling, (2) pre‑training, (3) fine‑tuning and alignment, (4) prompt engineering, and (5) evaluation. For each stage they define a concrete pitfall:

1. Data Poisoning (P1) – massive web‑scraped corpora can be silently poisoned.
2. LLM‑generated Label Inaccuracy (P2) – using LLMs as annotators introduces systematic labeling errors.
3. Data Leakage (P3) – overlap between pre‑training data and test/evaluation sets inflates performance.
4. Model Collapse via Synthetic Training Data (P4) – repeatedly fine‑tuning on self‑generated data reduces diversity and raises perplexity.
5. Spurious Correlations (P5) – LLMs may memorize non‑causal patterns, leading to shortcut learning.
6. Context Truncation (P6) – token‑window limits cause essential code or context to be dropped, especially in vulnerability detection.
7. Prompt Sensitivity (P7) – minor wording changes or model‑specific prompt preferences cause large output variance.
8. Proxy/Surrogate Fallacy (P8) – a model name (e.g., “ChatGPT”) often hides multiple underlying snapshots, architectures, or quantization levels, leading to misleading conclusions.
9. Model Ambiguity (P9) – insufficient specification of version, quantization, or tokenizer makes exact replication impossible.

To assess how pervasive these issues are, the authors performed a systematic prevalence study on 72 peer‑reviewed papers published between January 2023 and December 2024 at top security (IEEE S&P, NDSS, ACM CCS, USENIX Security) and software‑engineering venues (ICSE, FSE, ISSTA, ASE). A team of 15 researchers independently labeled each paper using a four‑level rubric (Present, Partially Present, Unclear, Not Present). The findings are striking: every single paper exhibited at least one of the nine pitfalls, and five pitfalls (P3, P6, P7, P8, P9) appeared in more than 20 % of the papers. Papers focused on vulnerability detection/repair showed the highest average pitfall density (≈ 23‑28 %), while fuzzing and secure code generation papers were slightly lower (≈ 15‑18 %).

Beyond prevalence, the authors conducted four empirical case studies to demonstrate concrete impact:

• Model Ambiguity – Using different snapshots or quantization levels of the same named model caused precision/recall variations of up to ±15 %, undermining reproducibility.
• Data Leakage – Injecting 20 % of test data into fine‑tuning raised F1 scores by 0.08‑0.11; the effect grew roughly linearly with leakage proportion.
• Context Truncation – 49 % of vulnerable functions exceeded a 512‑token window (29 % exceeded 1024 tokens); truncation led to a 30 % drop in detection accuracy.
• Model Collapse – Recursive self‑training on synthetic code increased perplexity by 1.8× after five generations and reduced code‑generation accuracy by 22 %.

These experiments confirm that the identified pitfalls are not merely theoretical concerns; they can materially inflate performance metrics, hide security weaknesses, and cripple the ability of other researchers to replicate findings.

The paper concludes with actionable guidelines for each pitfall. Recommendations include:

• Implement automated cross‑checking between pre‑training corpora and evaluation sets to detect leakage.
• Use human‑in‑the‑loop verification or multi‑annotator consensus when LLMs generate labels.
• Adopt sliding‑window or summarization techniques to mitigate context truncation.
• Conduct prompt‑sensitivity analyses (e.g., multiple paraphrases, ensemble prompting) and publish the full prompt set.
• Explicitly record model metadata—snapshot ID, quantization level, tokenizer version, API access date—and attach a DOI or Git commit hash to the experimental artifact.
• Maintain a living “LLM Pitfalls” repository (https://llmpitfalls.org) with up‑to‑date best‑practice checklists.

All code, datasets, and reproducibility instructions are released on GitHub (https://github.com/dormant-neurons/llm-pitfalls). The authors stress that the goal is not to blame researchers; rather, the rapid adoption of LLMs has outpaced methodological safeguards, and raising awareness is essential for the community’s long‑term credibility.

In sum, the study provides a comprehensive taxonomy of LLM‑specific research hazards, quantifies their prevalence in recent top‑tier security literature, empirically validates their impact, and supplies concrete mitigation strategies—an indispensable resource for anyone conducting rigorous, reproducible LLM‑based security research.