The Semantic Trap: Do Fine-tuned LLMs Learn Vulnerability Root Cause or Just Functional Pattern?

LLMs demonstrate promising performance in software vulnerability detection after fine-tuning. However, it remains unclear whether these gains reflect a genuine understanding of vulnerability root causes or merely an exploitation of functional patterns. In this paper, we identify a critical failure mode termed the “semantic trap,” where fine-tuned LLMs achieve high detection scores by associating certain functional domains with vulnerability likelihood rather than reasoning about the underlying security semantics. To systematically evaluate this phenomenon, we propose TrapEval, a comprehensive evaluation framework designed to disentangle vulnerability root cause from functional pattern. TrapEval introduces two complementary datasets derived from real-world open-source projects: V2N, which pairs vulnerable code with unrelated benign code, and V2P, which pairs vulnerable code with its corresponding patched version, forcing models to distinguish near-identical code that differs only in subtle security-critical logic. Using TrapEval, we fine-tune five representative state-of-the-art LLMs across three model families and evaluate them under cross-dataset testing, semantic-preserving perturbations, and varying degrees of semantic gap measured by CodeBLEU. Our empirical results reveal that, despite improvements in metrics, fine-tuned LLMs consistently struggle to distinguish vulnerable code from its patched counterpart, exhibit severe robustness degradation under minor semantic-preserving transformations, and rely heavily on functional-context shortcuts when the semantic gap is small. These findings provide strong evidence that current fine-tuning practices often fail to impart true vulnerability reasoning. Our findings serve as a wake-up call: high benchmark scores on traditional datasets may be illusory, masking the model’s inability to understand the true causal logic of vulnerabilities.

💡 Research Summary

The paper investigates whether large language models (LLMs) that have been fine‑tuned on vulnerability detection data truly learn the underlying security semantics of bugs, or simply exploit superficial functional patterns in code. To expose this “semantic trap,” the authors construct two complementary datasets from real‑world open‑source projects. V2N (Vulnerable‑to‑Normal) pairs each vulnerable snippet with an unrelated benign snippet, reproducing the traditional binary classification setting. V2P (Vulnerable‑to‑Patch) pairs each vulnerable function with its patched version, forcing the model to distinguish near‑identical code that differs only in the security‑critical logic. By measuring the semantic distance between the two versions with CodeBLEU, the authors can stratify examples from almost identical (high CodeBLEU) to substantially rewritten (low CodeBLEU).

Using the TrapEval framework, the study fine‑tunes five state‑of‑the‑art LLMs from three families (Qwen, Llama, DeepSeek) via LoRA. Three research questions are addressed: (RQ1) How does fine‑tuning affect detection performance and how does the composition of training data (V2N vs. V2P) influence it? (RQ2) How robust are the models to semantic‑preserving perturbations such as renaming parameters, reformatting, or variable scope changes? (RQ3) How does the semantic gap between vulnerable and patched code affect model accuracy?

Results show that fine‑tuning does improve overall F1 scores on the standard V2N benchmark, but the improvement is brittle. When V2P‑fine‑tuned models are evaluated on V2N test data (and vice‑versa), performance collapses, indicating that the models have learned dataset‑specific functional cues rather than general security reasoning. Under semantic‑preserving transformations, accuracy drops by 5‑12 % on average, with some models losing more than 20 % of their predictive power, confirming a reliance on surface token patterns. Crucially, the analysis of CodeBLEU‑based semantic gaps reveals a strong correlation: for pairs with CodeBLEU > 0.95 (almost identical), detection accuracy falls below 30 %; for pairs with CodeBLEU < 0.75 (larger semantic changes), accuracy rises above 70 %. This demonstrates that models succeed only when the patch introduces noticeable functional differences, not when the fix is a subtle logical correction.

The authors argue that current fine‑tuning practices give a false sense of security: high benchmark scores mask a fundamental inability to reason about the root cause of vulnerabilities. They propose several avenues for future work: (1) redesigning labeling schemes to capture explicit security invariants (e.g., “loop invariant violation”), (2) incorporating semantic‑preserving augmentations during training to improve robustness, (3) developing interpretability tools (attention visualizations, gradient analysis) to verify that models attend to security‑relevant code regions, and (4) exploring multimodal training that combines code with static analysis reports.

In summary, the paper provides a rigorous methodology (TrapEval) and compelling empirical evidence that fine‑tuned LLMs are often trapped in functional pattern shortcuts rather than learning true vulnerability semantics. This work calls for a reevaluation of evaluation protocols and training strategies for LLM‑based security tools, emphasizing the need for semantics‑aware benchmarks and robustness testing before deployment in real‑world software assurance pipelines.