Following Dragons: Code Review-Guided Fuzzing

Modern fuzzers scale to large, real-world software but often fail to exercise the program states developers consider most fragile or security-critical. Such states are typically deep in the execution space, gated by preconditions, or overshadowed by lower-value paths that consume limited fuzzing budgets. Meanwhile, developers routinely surface risk-relevant insights during code review, yet this information is largely ignored by automated testing tools. We present EyeQ, a system that leverages developer intelligence from code reviews to guide fuzzing. EyeQ extracts security-relevant signals from review discussions, localizes the implicated program regions, and translates these insights into annotation-based guidance for fuzzing. The approach operates atop existing annotation-aware fuzzing, requiring no changes to program semantics or developer workflows. We first validate EyeQ through a human-guided feasibility study on a security-focused dataset of PHP code reviews, establishing a strong baseline for review-guided fuzzing. We then automate the workflow using a large language model with carefully designed prompts. EyeQ significantly improves vulnerability discovery over standard fuzzing configurations, uncovering more than 40 previously unknown bugs in the security-critical PHP codebase.

💡 Research Summary

The paper “Following Dragons: Code Review‑Guided Fuzzing” introduces EyeQ, a system that bridges the gap between developer intuition expressed in code‑review discussions and modern coverage‑guided fuzzing. Modern fuzzers such as AFL++, LibAFL, and Honggfuzz excel at scaling to large, real‑world codebases, but they often miss the most security‑critical states because those states lie deep in the program’s state graph, are gated by complex pre‑conditions, or are simply eclipsed by high‑frequency low‑value paths. The authors argue that developers routinely flag such risky code during code reviews—comments like “this function assumes a non‑null pointer” or “stack size validation is insufficient”—yet these insights are never fed back into automated testing pipelines.

EyeQ’s workflow consists of three stages: (1) Security‑relevant review extraction, (2) Program localization, and (3) Annotation generation for fuzzing. In the first stage, a large language model (LLM) is prompted to filter raw review comments for security relevance, using keywords and contextual cues derived from CWE taxonomies. The second stage combines static analysis (call‑graph construction, data‑flow analysis) with natural‑language‑code mapping to pinpoint the exact functions, variables, or conditionals referenced in the reviews. The third stage automatically inserts IJON‑style annotations (e.g., IJON_SET, IJON_MAX, IJON_MIN, IJON_STATE) at the identified locations. IJON is an annotation‑aware fuzzing framework that treats these programmer‑provided signals as first‑class feedback, allowing the fuzzer to prioritize executions that make semantic progress even when traditional edge coverage stalls.

The authors evaluate EyeQ on a large, security‑critical PHP codebase. First, a human‑guided feasibility study is performed: security‑focused reviewers manually annotate the code based on 41 review comments, and AFL++ equipped with IJON discovers 41 previously unknown bugs. This establishes a strong baseline. Next, the fully automated pipeline is run on the same codebase. The LLM‑driven extraction and annotation insertion yield 46 new bugs, surpassing the human baseline and the standard OSS‑Fuzz configuration by more than 30 %. The discovered vulnerabilities span stack overflows, unchecked input handling, and memory‑safety violations, many of which require precise numeric values (e.g., malformed zend.stack_size) that conventional coverage‑guided fuzzers rarely generate.

Key contributions include: (C1) recognizing code reviews as an under‑exploited source of security intelligence; (C2) proposing a generic, annotation‑based translation of unstructured review text into actionable fuzzing guidance; (C3) implementing both a human‑guided proof‑of‑concept and a fully automated LLM‑driven system; and (C4) empirically demonstrating that review‑guided fuzzing can uncover dozens of critical bugs missed by state‑of‑the‑art fuzzers.

The paper also discusses practical considerations. EyeQ operates atop existing CI pipelines without requiring developers to write additional annotations; the only prerequisite is the presence of code‑review data. Limitations include dependence on the quality and coverage of reviews, potential false positives in LLM‑based extraction, and the current focus on PHP (future work should test other languages and larger ecosystems). The authors suggest extending the approach to multi‑language settings, improving the precision of the natural‑language‑to‑code mapping, and exploring hybrid feedback that combines annotation signals with richer dynamic analyses.

In summary, EyeQ showcases a compelling synergy between human insight and automated testing: by automatically converting developer‑authored review comments into semantic guidance for an annotation‑aware fuzzer, it dramatically improves vulnerability discovery in real‑world software, offering a practical path toward more secure software development pipelines.