Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Framework

Visualizations play a crucial part in effective communication of concepts and information. Recent advances in reasoning and retrieval augmented generation have enabled Large Language Models (LLMs) to perform deep research and generate comprehensive reports. Despite its progress, existing deep research frameworks primarily focus on generating text-only content, leaving the automated generation of interleaved texts and visualizations underexplored. This novel task poses key challenges in designing informative visualizations and effectively integrating them with text reports. To address these challenges, we propose Formal Description of Visualization (FDV), a structured textual representation of charts that enables LLMs to learn from and generate diverse, high-quality visualizations. Building on this representation, we introduce Multimodal DeepResearcher, an agentic framework that decomposes the task into four stages: (1) researching, (2) exemplar report textualization, (3) planning, and (4) multimodal report generation. For the evaluation of generated multimodal reports, we develop MultimodalReportBench, which contains 100 diverse topics served as inputs along with 5 dedicated metrics. Extensive experiments across models and evaluation methods demonstrate the effectiveness of Multimodal DeepResearcher. Notably, utilizing the same Claude 3.7 Sonnet model, Multimodal DeepResearcher achieves an 82% overall win rate over the baseline method.

💡 Research Summary

Introduction and Problem Statement The rapid evolution of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) has significantly enhanced the capabilities of AI in conducting deep research and generating comprehensive textual reports. However, a critical gap remains in the current landscape of automated research frameworks: the lack of multimodal integration. While professional reports, scientific papers, and business documents rely heavily on the seamless interleaving of text and visualizations (charts, graphs, etc.), existing deep research agents primarily focus on text-only outputs. The challenge lies in the complex task of designing informative visualizations and logically integrating them with textual narratives to create a cohesive, high-quality multimodal report.

Proposed Solution: Multimodal DeepResearcher To bridge this gap, the paper introduces Multimodal DeepResearcher, an innovative agentic framework designed to generate text-chart interleaved reports from scratch. The core technical breakthrough of this research is the Formal Description of Visualization (FDV). FDV is a structured textual representation of charts that translates complex visual elements into a format that LLMs can effectively process, learn from, and generate. By treating visualizations as structured text, the framework enables the LLM to reason about the data mapping and structural components of a chart, ensuring high-quality and accurate visual outputs.

The framework operates through a sophisticated four-stage agentic pipeline:

1. Researching: The agent conducts deep information retrieval to gather all necessary data related to the research topic.
2. Exemplar Report Textualization: The system analyzes high-quality exemplar reports to learn the patterns of how text and charts should be integrated.
3. Planning: Based on the researched data, the agent creates a structural blueprint, determining the report’s hierarchy and deciding the optimal placement for each visualization.
4. Multimodal Report Generation: The final stage assembles the text and the FDV-based charts into a complete, interleaved multimodal document.

Evaluation and Experimental Results To rigorously assess the performance of the generated reports, the authors developed MultimodalReportBench, a new benchmark consisting of 100 diverse research topics and 5 dedicated metrics designed to evaluate the quality of multimodal integration. The experiments were conducted across various models, with a particular focus on the Claude 3.7 Sonnet model. The results were highly significant, demonstrating that Multimodal DeepResearcher achieved an 82% overall win rate over the baseline text-only methods.

Conclusion and Impact The Multimodal DeepResearcher represents a major step forward in the field of autonomous AI agents. By introducing FDV and a structured multi-stage workflow, the researchers have demonstrated that AI can move beyond simple text generation to become a sophisticated creator of complex, multimodal documents. This advancement holds immense potential for automating professional-grade reporting in industries such as finance, science, and business, where the synergy between data visualization and textual analysis is paramount.