Vital: Vulnerability-Oriented Symbolic Execution via Type-Unsafe Pointer-Guided Monte Carlo Tree Search

How to find memory safety bugs efficiently when navigating a symbolic execution tree that suffers from path explosion? Existing solutions either adopt path search heuristics to maximize coverage rate or chopped symbolic execution to skip uninteresting code (i.e., manually labeled as vulnerability-unrelated) during path exploration. However, most existing search heuristics are not vulnerability-oriented, and manual labeling of irrelevant code-to-be-skipped relies heavily on prior expert knowledge, making it hard to detect vulnerabilities effectively in practice. This paper proposes Vital, a new vulnerability-oriented path exploration for symbolic execution with two innovations. First, a new indicator (i.e., type-unsafe pointers) is suggested to approximate vulnerable paths. A pointer that is type-unsafe cannot be statically proven to be safely dereferenced without memory corruption. Our key hypothesis is that a path with more type-unsafe pointers is more likely to be vulnerable. Second, a new type-unsafe pointer-guided Monte Carlo Tree Search algorithm is implemented to guide the path exploration towards the areas that contain more unsafe pointers, aiming to increase the likelihood of detecting vulnerabilities. We built Vital on top of KLEE and compared it with existing path searching strategies and chopped symbolic execution. In the former, the results demonstrate that Vital could cover up to 90.03% more unsafe pointers and detect up to 57.14% more unique memory errors. In the latter, the results show that Vital could achieve a speedup of up to 30x execution time and a reduction of up to 20x memory consumption to detect known vulnerabilities without prior expert knowledge automatically. In practice, Vital also detected one previously unknown vulnerability (a new CVE ID is assigned), which has been fixed by developers.

💡 Research Summary

The paper introduces Vital, a vulnerability‑oriented path exploration strategy for symbolic execution that tackles the classic path‑explosion problem by focusing on memory‑safety bugs. The authors first propose a novel indicator—type‑unsafe pointers—to approximate how “vulnerable” a program path is. A type‑unsafe pointer is one that cannot be statically proven safe (i.e., classified as SEQ or DYN by tools such as CCured). Because spatial memory errors (buffer overflows, out‑of‑bounds accesses) can only occur when dereferencing such pointers, the number of type‑unsafe pointers exercised along a path serves as a proxy for its bug‑proneness. Empirical data from the authors’ experiments show a Pearson correlation of 0.93 between the count of unsafe pointers and the number of detected memory errors, validating the hypothesis.

To exploit this indicator, Vital integrates a Monte Carlo Tree Search (MCTS) algorithm into the KLEE symbolic executor. Traditional MCTS consists of selection, expansion, simulation, and back‑propagation, using a tree‑policy (often Upper Confidence Bound for Trees, UCT) to balance exploration and exploitation. Vital modifies the reward function so that nodes with higher counts of type‑unsafe pointers receive larger rewards. During selection, the UCT formula is adjusted to prioritize sub‑trees that have historically exhibited many unsafe pointers, thereby steering the search toward regions of the program that are more likely to contain memory‑safety violations. This approach differs from existing heuristics (BFS, DFS, random, coverage‑guided) which aim at code coverage without explicit regard for vulnerability likelihood.

Implementation details: type‑unsafe pointer identification is performed at the LLVM‑IR level. Whenever a pointer operation (arithmetic, cast, dereference) appears, metadata marking the pointer as SAFE, SEQ, or DYN is attached. This metadata is updated dynamically as symbolic execution proceeds, allowing the MCTS component to query the current “unsafe‑pointer count” for any state. The MCTS logic is woven into KLEE’s state‑selection loop with minimal intrusion, preserving KLEE’s existing constraint‑solving pipeline.

The evaluation comprises two parts. First, the authors benchmark Vital against six standard KLEE search strategies (DFS, BFS, Random, nurs:covnew, nurs:icnt, etc.) and the recent CBC strategy on the GNU Coreutils suite. Vital achieves up to 90 % higher coverage of unsafe pointers and discovers up to 57 % more unique memory errors than the best baseline. Second, they test six real‑world CVE‑containing programs, comparing Vital with vanilla KLEE, Chopper (a chopped symbolic execution framework), and CBC. Vital automatically finds all known CVEs while achieving up to 30× speed‑up and 20× reduction in memory consumption relative to the baselines. Notably, Vital also uncovers a previously unknown vulnerability (assigned CVE‑2025‑3198), which was subsequently patched by the developers, demonstrating practical impact.

The choice of type‑unsafe pointers as a vulnerability metric offers two major advantages. First, static type inference is inexpensive and can be integrated into existing symbolic execution pipelines without heavy instrumentation. Second, the metric directly correlates with spatial memory safety, the most exploitable class of bugs in C/C++ code. However, the approach has limitations: it does not directly address temporal memory errors such as use‑after‑free, and its effectiveness depends on the accuracy of the underlying type‑inference analysis. Optimizations that obscure pointer types (e.g., aggressive compiler transformations) could lead to mis‑classification and sub‑optimal rewards.

The authors discuss future work to mitigate these issues. Potential extensions include augmenting the reward function with additional static or dynamic signals (e.g., potential overflow arithmetic, disparity between allocation size and actual usage) and combining MCTS with reinforcement‑learning policies that can generalize across programs. They also suggest exploring hybrid strategies that blend type‑unsafe pointer guidance with traditional coverage‑guided heuristics to capture a broader spectrum of bugs.

In summary, Vital demonstrates that a focused, vulnerability‑oriented search guided by a simple yet powerful indicator can dramatically improve the efficiency and effectiveness of symbolic execution for memory‑safety bug discovery. By marrying type‑unsafe pointer analysis with a tailored Monte Carlo Tree Search, the system reduces the need for expert‑provided “chop” lists, accelerates bug detection, and even uncovers previously unknown security flaws, offering a compelling direction for next‑generation automated vulnerability analysis tools.