Efficient Table Retrieval and Understanding with Multimodal Large Language Models

Tabular data is frequently captured in image form across a wide range of real-world scenarios such as financial reports, handwritten records, and document scans. These visual representations pose unique challenges for machine understanding, as they combine both structural and visual complexities. While recent advances in Multimodal Large Language Models (MLLMs) show promising results in table understanding, they typically assume the relevant table is readily available. However, a more practical scenario involves identifying and reasoning over relevant tables from large-scale collections to answer user queries. To address this gap, we propose TabRAG, a framework that enables MLLMs to answer queries over large collections of table images. Our approach first retrieves candidate tables using jointly trained visual-text foundation models, then leverages MLLMs to perform fine-grained reranking of these candidates, and finally employs MLLMs to reason over the selected tables for answer generation. Through extensive experiments on a newly constructed dataset comprising 88,161 training and 9,819 testing samples across 8 benchmarks with 48,504 unique tables, we demonstrate that our framework significantly outperforms existing methods by 7.0% in retrieval recall and 6.1% in answer accuracy, offering a practical solution for real-world table understanding tasks.

💡 Research Summary

The paper introduces TabRAG, an end‑to‑end framework that enables Multimodal Large Language Models (MLLMs) to answer user queries over massive collections of table images. The authors observe that most existing table‑understanding work assumes the target table is already provided, which is unrealistic in real‑world settings where users must locate the relevant table among thousands of scanned or rendered tables. TabRAG addresses this gap with a three‑stage pipeline: retrieval, reranking, and generation.

In the retrieval stage, a bi‑encoder architecture jointly fine‑tunes a visual encoder (hα) and a text encoder (gβ) on paired (query, table‑image) data using a contrastive loss that maximizes cosine similarity for matching pairs while pushing apart non‑matching pairs. After training, all table images in the datastore are encoded offline, and FAISS is used for fast approximate nearest‑neighbor search. Given a textual query q, its embedding gβ(q) is compared to the pre‑computed image embeddings, and the top‑n most similar tables are returned.

The reranking stage replaces simple vector similarity with a cross‑encoder MLLM (fθ). Each candidate image is concatenated with the query in a binary prompt asking “Is this image relevant to the question?” The model’s probability of outputting “True” is used to rank the candidates. Training for this stage mixes positive pairs (query + ground‑truth table) with hard negatives sampled from the top‑retrieved but irrelevant tables, optimizing a binary classification loss. This allows the system to capture fine‑grained visual‑textual cues that the bi‑encoder cannot.

In the generation stage, the top‑k reranked tables are fed together with the query into the same MLLM. The prompt instructs the model to produce the final answer in a structured JSON format, effectively performing retrieval‑augmented generation with multimodal context. The authors convert image embeddings into token sequences so that the LLM can attend to multiple images simultaneously.

To evaluate TabRAG, the authors built a new multimodal dataset by converting 14 public table corpora (MMTab) into high‑quality images, then filtering queries to focus on information‑seeking questions rather than generic table operations. The final dataset contains 88,161 training and 9,819 test samples, covering 48,504 unique tables across eight domains (finance, healthcare, etc.). Experiments measure Recall@k for retrieval, accuracy for reranking, and Exact Match/F1 for final QA. TabRAG outperforms strong baselines—including BM25, DPR, and recent multimodal retrieval models—by 7.0 percentage points in recall and 6.1 points in answer accuracy. Gains are especially pronounced for tables with merged cells, colored highlights, or embedded images, where visual structure matters.

Key contributions are: (1) a jointly trained visual‑text bi‑encoder that enables fast, OCR‑free search over table images; (2) the use of an MLLM as both a fine‑grained reranker and a multimodal generator, introducing a novel “visual‑language relevance” capability; (3) the construction of a large, realistic multimodal table QA benchmark. The work demonstrates that integrating retrieval and generation within a single multimodal LLM pipeline can bridge the gap between textual queries and visual table data, offering a practical solution for real‑world applications such as financial report analysis, medical record lookup, and large‑scale document digitization.