WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements

Visual language model (VLM) agents show great promise in automating end-to-end (E2E) web testing against requirements in natural language. However, the probabilistic nature of language models can have inherent hallucinations. Therefore, given a detected inconsistency between the requirement and the web application, it is hard to distinguish whether it stems from the hallucination or a real application bug. Addressing this issue presents two core technical challenges: the implicit oracle inference challenge, where the agent must act as its own oracle to implicitly decide if the application’s behavior is correct without guidance, and the probabilistic inference challenge, where an LLM’s inconsistent reasoning undermines its trustworthiness as an oracle. Existing LLM-based approaches fail to capture such implicit oracles, either by treating any page navigation that doesn’t crash as a success, or by checking each state in isolation, thus missing bugs dependent on context from prior steps. We introduce WebTestPilot, an LLM-based agent designed to address these challenges. WebTestPilot uses (1) a symbolization layer which detects and symbolizes critical GUI elements on the web application into symbols (i.e., variables) and (2) translates natural language specification into a sequence of steps, each of which is equipped with inferred pre- and post-conditions over the symbols as an oracle. This oracle captures data, temporal, and causal dependencies, enabling the validation of implicit requirements. To advance research in this area, we build a benchmark of bug-injected web apps for evaluating NL-to-E2E testing. The results show that WebTestPilot achieves a task completion rate of 99%, with 96% precision and 96% recall in bug detection, outperforming the best baseline (+70 precision, +27 recall). The agent generalizes across diverse natural language inputs and model scales.

💡 Research Summary

WebTestPilot tackles two fundamental obstacles that have limited the reliability of large‑language‑model (LLM) agents for end‑to‑end (E2E) web testing: (1) the implicit‑oracle problem, where many correctness criteria are not explicitly stated in the specification but depend on the history of user actions, and (2) the probabilistic‑inference problem, where the stochastic nature of LLMs yields inconsistent reasoning across test steps. Existing tools such as NaïveQAte, LaVague, and PinATA either treat any navigation that does not crash as success or verify each UI state in isolation, thereby missing bugs that manifest only when cross‑state dependencies are considered.

WebTestPilot introduces a two‑layer architecture to overcome these challenges. The first layer, called the symbolization layer, processes screenshots of the web application, detects critical GUI elements (buttons, labels, input fields, etc.) using OCR and visual object detection, and maps each element to a symbolic variable. These symbols are instantiated through strongly‑typed schemas (implemented with Pydantic models) that capture data types, required fields, and domain constraints (e.g., non‑negative price). By maintaining a history buffer of past symbols, the system can extract three kinds of dependencies across states: data (values that must be consistent, such as product title and price), temporal (changes observed when revisiting the same logical page), and causal (actions that must precede others, such as focusing the search bar before typing).

The second layer translates a natural‑language requirement into a sequence of (condition, action, expectation) triples. For each triple WebTestPilot automatically generates formal pre‑ and post‑conditions expressed in a domain‑specific language (DSL). The DSL allows logical assertions such as “cart.items = prior.items ∪ {new_product} ∧ new_product.title = viewed_product.title ∧ new_product.quantity = 1”. These assertions constitute a declarative oracle that can be evaluated against the current UI state and the stored symbolic history.

To mitigate LLM stochasticity, WebTestPilot employs a multi‑candidate strategy. For any given step it asks the LLM to produce m candidate assertions, then evaluates each candidate against the actual UI. A majority‑vote or a configurable threshold determines whether the step passes. This “re‑trial” mechanism statistically smooths out hallucinations and yields a stable oracle even when the underlying model is probabilistic.

Implementation-wise, WebTestPilot builds on existing automation frameworks (Selenium/Playwright). After each action, the agent captures a screenshot, runs the symbolizer, updates the symbolic history, and invokes the DSL interpreter to check the generated assertions. If a check fails, the agent records a bug report that includes the violated logical formula and the visual evidence.

The authors constructed a benchmark consisting of four open‑source web applications into which they injected 110 bugs of various types (data‑binding, UI layout, navigation, and state‑consistency errors). Compared with the three baselines, WebTestPilot achieved a 99 % test‑completion rate and 96 % precision and recall in bug detection. The advantage was especially pronounced for scenarios that required reasoning about implicit dependencies; the baselines lagged by up to 70 % in precision and 27 % in recall. Moreover, the approach remained robust across model sizes ranging from 3 B to 72 B parameters and across noisy natural‑language inputs containing typos, grammatical errors, redundant sentences, stylistic restyling, or abbreviations.

A real‑world deployment with a no‑code platform partner demonstrated practical impact: during development, WebTestPilot uncovered eight distinct bugs—covering data‑binding mismatches, UI rendering glitches, and navigation failures—that had escaped manual testing.

The paper’s contributions are fourfold: (1) a novel methodology for inferring implicit oracles from arbitrary natural‑language specifications via GUI symbolization and DSL‑based pre/post‑conditions; (2) a concrete system implementation that integrates LLM reasoning with symbolic verification; (3) a publicly released benchmark of bug‑injected web apps to foster further research; and (4) extensive empirical validation showing both high effectiveness and scalability. By bridging neural language understanding with formal symbolic reasoning, WebTestPilot sets a new standard for trustworthy, automated web testing and opens avenues for applying similar techniques to broader software verification tasks.