DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents

We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent’s ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.

💡 Research Summary

DeepSearchQA is a newly introduced benchmark designed to evaluate the “deep research” capabilities of autonomous web agents. Unlike traditional QA datasets that focus on single‑answer factual retrieval, DeepSearchQA comprises 900 handcrafted, multi‑step prompts spanning 17 domains (e.g., epidemiology, finance, gaming, safety, demography). Each prompt is a causal chain: the answer to step n is required to formulate the query for step n + 1, thereby testing long‑horizon planning, context retention, and the ability to synthesize information from disparate web sources.

The authors identify three under‑evaluated capabilities that current agents lack: (1) Systematic Collation – the ability to visit hundreds of heterogeneous pages, extract partial facts, and merge them into a master list; (2) Entity Resolution (De‑duplication) – recognizing that differently worded mentions refer to the same real‑world entity; and (3) Stopping Criteria – deciding when the search space is exhausted without an explicit termination signal, balancing epistemic uncertainty against computational cost.

Dataset construction involved expert annotators, a three‑phase verification pipeline (independent research, cross‑validation, conflict resolution), and a strict focus on objective, time‑anchored data sources to avoid ground‑truth drift. Answers are either single‑entity values or sets (enumerations or composite responses). Evaluation is outcome‑centric: agents are scored solely on the completeness (recall) and correctness (precision) of the final answer set, using an F1‑based metric that disregards ordering. This design encourages agents to master the exploration‑exploitation trade‑off rather than merely optimizing for a single correct fact.

Experiments benchmarked several state‑of‑the‑art agents, including Google’s Gemini Deep Research Agent, Anthropic’s Claude, and OpenAI’s GPT‑4o. Across the full suite, average recall was ~0.62 and precision ~0.48, yielding an overall F1 of ~0.55. Performance dropped sharply on “hard” prompts that required deeper dependency graphs, with recall falling below 0.45. Failure mode analysis revealed two dominant patterns: premature stopping (agents halt before gathering all relevant items) and hedging (agents over‑generate low‑confidence candidates, inflating recall but harming precision). These patterns expose a fundamental limitation: current agents lack robust mechanisms for cost‑aware search planning and for principled uncertainty estimation.

The paper proposes a new scoring schema that jointly measures set‑level recall and precision, and it releases a live Kaggle leaderboard with automated verification to foster community‑driven progress. By exposing the “comprehensiveness gap,” DeepSearchQA establishes a diagnostic tool that pushes research toward three concrete directions: (1) learning efficient, multi‑step search policies (e.g., reinforcement‑learning‑based exploration); (2) building sophisticated entity resolution pipelines that combine lexical similarity, contextual embeddings, and external knowledge bases; and (3) integrating epistemic uncertainty models to decide optimal stopping points.

In summary, DeepSearchQA shifts the evaluation paradigm from precision‑centric single‑answer retrieval to exhaustive answer‑set generation, revealing that even the most advanced LLM‑based agents struggle to balance recall and precision in open‑ended, multi‑domain research tasks. The benchmark’s rich taxonomy (Structured Retrieval, Context Management, Logical Reasoning) and its rigorous verification protocol make it a valuable resource for guiding the next generation of deep‑research agents.