SPECA: Specification-to-Checklist Agentic Auditing for Multi-Implementation Systems -- A Case Study on Ethereum Clients

Multi-implementation systems are increasingly audited against natural-language specifications. Differential testing scales well when implementations disagree, but it provides little signal when all implementations converge on the same incorrect interpretation of an ambiguous requirement. We present SPECA, a Specification-to-Checklist Auditing framework that turns normative requirements into checklists, maps them to implementation locations, and supports cross-implementation reuse. We instantiate SPECA in an in-the-wild security audit contest for the Ethereum Fusaka upgrade, covering 11 production clients. Across 54 submissions, 17 were judged valid by the contest organizers. Cross-implementation checks account for 76.5 percent (13 of 17) of valid findings, suggesting that checklist-derived one-to-many reuse is a practical scaling mechanism in multi-implementation audits. To understand false positives, we manually coded the 37 invalid submissions and find that threat model misalignment explains 56.8 percent (21 of 37): reports that rely on assumptions about trust boundaries or scope that contradict the audit’s rules. We detected no High or Medium findings in the V1 deployment; misses concentrated in specification details and implicit assumptions (57.1 percent), timing and concurrency issues (28.6 percent), and external library dependencies (14.3 percent). Our improved agent, evaluated against the ground truth of a competitive audit, achieved a strict recall of 27.3 percent on high-impact vulnerabilities, placing it in the top 4 percent of human auditors and outperforming 49 of 51 contestants on critical issues. These results, though from a single deployment, suggest that early, explicit threat modeling is essential for reducing false positives and focusing agentic auditing effort. The agent-driven process enables expert validation and submission in about 40 minutes on average.

💡 Research Summary

The paper introduces SPECA (Specification‑to‑Checklist Auditing), a framework designed to bridge the gap between natural‑language specifications and the code of multiple independent implementations. Traditional approaches—differential testing and formal verification—either rely on behavioral divergence as an oracle or demand a full formalization of the specification, both of which are inadequate when many implementations share the same misinterpretation of an ambiguous requirement (the “semantic blind spot”). SPECA tackles this by converting normative requirements into a structured, traceable checklist, mapping each item to concrete locations in the source code of every client, and then reusing those checks across implementations.

SPECA’s workflow consists of two main phases. Phase 1 (Knowledge Structuring) uses a large‑language‑model (LLM) extractor to locate RFC 2119‑style statements (MUST, SHOULD, etc.) in specification documents, assign unique identifiers, and store them in a knowledge base. It also builds a pattern database of known vulnerability motifs (e.g., boundary checks, cryptographic misuse) and creates an implementation‑mapping layer that first prunes candidate code locations with keyword search and then refines the mapping via LLM‑driven semantic analysis. Phase 2 (Systematic Auditing) applies three complementary strategies. Strategy A performs direct static or symbolic checks against each implementation to verify presence, correctness, and completeness of the checklist items. Strategy B, the core contribution, propagates a failure discovered in one client to all other clients: it abstracts the offending pattern, locates analogous code fragments in the remaining implementations, and re‑evaluates them. This “1 → N” reuse directly addresses semantic blind spots because correlated misinterpretations become visible even when all clients behave identically. A third strategy (human‑in‑the‑loop validation) ensures that every reported violation is vetted by an expert.

The authors evaluated SPECA in a real‑world security‑audit contest for the Ethereum “Fusaka” upgrade, covering eleven production clients. Out of 54 submissions, 17 were deemed valid by the contest organizers. Remarkably, 13 of those 17 (76.5 %) were discovered via cross‑implementation checks (Strategy B), demonstrating that checklist‑driven reuse is a practical scaling mechanism in multi‑implementation audits. To understand the 37 invalid submissions, the authors manually coded them and found that 56.8 % (21 reports) suffered from threat‑model misalignment—reporters assumed attacker capabilities or trust boundaries that conflicted with the contest’s explicit rules. This highlights that, beyond a solid checklist, an explicitly shared threat model is essential to curb false positives.

Further analysis of the V1 deployment (the baseline audit) revealed no high‑ or medium‑severity findings; the missed issues clustered in three categories: specification details and implicit assumptions (57.1 %), timing and concurrency problems (28.6 %), and external library dependencies (14.3 %). These categories are precisely the kinds of defects that are hard to capture with pure static analysis or fuzzing, underscoring the value of a specification‑anchored approach.

The improved SPECA agent (V2) was benchmarked against a “ground‑truth” audit (the Sherlock “Can’t‑be‑evil” dataset). On consensus‑layer issues rated Low severity or higher, the agent achieved a strict recall of 27.3 % (3 out of 11), including two high‑severity bugs missed by the earlier version. This performance placed the agent in the top 4 % of human auditors and outperformed 49 of 51 contestants on critical issues. Moreover, the agent‑driven pipeline reduced human effort dramatically: experts could validate a core finding in roughly 10 minutes and produce a full proof‑of‑concept and report in an additional 30 minutes, for an average total of 40 minutes per submission.

In summary, SPECA demonstrates that (1) turning normative specifications into reusable, traceable checklists enables systematic, cross‑client verification; (2) 1 → N checklist reuse is an effective antidote to semantic blind spots in multi‑implementation environments; (3) explicit threat‑model formalization is a decisive factor in minimizing false positives; and (4) an LLM‑augmented, artifact‑centric pipeline can achieve audit quality comparable to top human experts while substantially lowering manual labor. The authors suggest future work on automated threat‑model extraction, deeper integration of timing/concurrency analysis, and handling of external dependencies to broaden SPECA’s applicability across other protocol ecosystems.