VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications

Large language models (LLMs) have achieved remarkable progress in code understanding tasks. However, they demonstrate limited performance in vulnerability detection and struggle to distinguish vulnerable code from patched code. We argue that LLMs lack understanding of security specifications – the expectations about how code should behave to remain safe. When code behavior differs from these expectations, it becomes a potential vulnerability. However, such knowledge is rarely explicit in training data, leaving models unable to reason about security flaws. We propose VulInstruct, a specification-guided approach that systematically extracts security specifications from historical vulnerabilities to detect new ones. VulInstruct constructs a specification knowledge base from two perspectives: (i) General specifications from high-quality patches across projects, capturing fundamental safe behaviors; and (ii) Domain-specific specifications from repeated violations in particular repositories relevant to the target code. VulInstruct retrieves relevant past cases and specifications, enabling LLMs to reason about expected safe behaviors rather than relying on surface patterns. We evaluate VulInstruct under strict criteria requiring both correct predictions and valid reasoning. On PrimeVul, VulInstruct achieves 45.0% F1-score (32.7% improvement) and 37.7% recall (50.8% improvement) compared to baselines, while uniquely detecting 24.3% of vulnerabilities – 2.4x more than any baseline. In pair-wise evaluation, VulInstruct achieves 32.3% relative improvement. VulInstruct also discovered a previously unknown high-severity vulnerability (CVE-2025-56538) in production code, demonstrating practical value for real-world vulnerability discovery. All code and supplementary materials are available at https://github.com/zhuhaopku/VulInstruct-temp.

💡 Research Summary

The paper “VulInstruct: Teaching LLMs Root‑Cause Reasoning for Vulnerability Detection via Security Specifications” addresses a fundamental weakness of current large language models (LLMs) in software vulnerability detection: the lack of explicit security specifications. A security specification is an implicit expectation—defined by developers and security experts—about how code should behave safely. Because such expectations are rarely documented in source code or public resources, LLMs trained on massive code corpora cannot reason about the root causes of vulnerabilities; they tend to rely on surface‑level patterns and textual similarity.

VulInstruct proposes a specification‑guided framework that systematically extracts reusable security specifications from historical vulnerabilities and uses them to instruct LLMs when analyzing new code. The approach consists of two complementary automatic pipelines:

1. General Security Specifications – These are mined from high‑quality patches across a wide range of open‑source projects. By diffing vulnerable and patched versions, the system identifies the underlying expected safe behavior and restates it in natural language. Contextual information such as called functions, type declarations, imported modules, and global variables is also captured to produce a richer abstraction of the developer’s intent.

2. Domain‑Specific Security Specifications – These are derived from a comprehensive CVE database, focusing on vulnerabilities that repeatedly appear within the same repository or within a specific domain (e.g., networking libraries, web frameworks). By analyzing the patterns of repeated exploitation, the pipeline extracts expectations that are particularly relevant to the target codebase.

The two specification sets are stored in a knowledge base. When a new code snippet is presented, VulInstruct first retrieves the most similar historical vulnerability cases and their associated specifications. These specifications are then injected into the prompt given to an LLM (e.g., GPT‑4 or a fine‑tuned code model). The model is thus guided to reason about whether the code violates any of the expected safe behaviors, rather than merely matching known vulnerable patterns.

Evaluation is performed under the strict CORRECT framework, which requires both correct binary classification and a valid reasoning trace. On the PrimeVul benchmark—a widely used dataset that previously revealed less than 12 % accuracy for state‑of‑the‑art LLMs—VulInstruct achieves a 45.0 % F1 score, a 32.7 % relative improvement over the strongest baseline, and a 37.7 % recall (50.8 % improvement). Notably, 24.3 % of the vulnerabilities detected by VulInstruct are unique, representing a 2.4× gain over any baseline. In pair‑wise evaluation (distinguishing vulnerable code from its patched counterpart), the method improves accuracy by 32.3 % relative to the best prior approach.

Beyond benchmark results, the authors present a real‑world case study: applying VulInstruct to production code uncovered a previously unknown high‑severity vulnerability, later assigned CVE‑2025‑56538. The vulnerability corresponded to a violation of an extracted specification that had been observed in earlier CVEs, demonstrating that the specification‑guided reasoning can surface novel security flaws that pattern‑based methods miss.

Key contributions include:

• Introducing the concept of security specifications as a bridge between implicit developer knowledge and LLM reasoning.
• Designing a dual‑layer retrieval system that supplies both general and domain‑specific specifications to the model.
• Demonstrating that natural‑language specifications, automatically generated via LLM prompting, are more accessible and transferable than formal rule‑based representations.
• Providing a rigorous evaluation that combines accuracy with reasoning correctness, setting a higher standard for future LLM‑based security tools.

The paper concludes with a discussion of limitations (e.g., potential noise in automatically generated specifications, handling conflicting specifications) and outlines future directions such as specification refinement, multi‑language extension, and integration with static analysis pipelines. Overall, VulInstruct showcases how enriching LLMs with explicit security expectations can dramatically improve both detection performance and practical utility in software security.