The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems

We present the setup and the tasks of the FinMMEval Lab at CLEF 2026, which introduces the first multilingual and multimodal evaluation framework for financial Large Language Models (LLMs). While recent advances in financial natural language processing have enabled automated analysis of market reports, regulatory documents, and investor communications, existing benchmarks remain largely monolingual, text-only, and limited to narrow subtasks. FinMMEval 2026 addresses this gap by offering three interconnected tasks that span financial understanding, reasoning, and decision-making: Financial Exam Question Answering, Multilingual Financial Question Answering (PolyFiQA), and Financial Decision Making. Together, these tasks provide a comprehensive evaluation suite that measures models’ ability to reason, generalize, and act across diverse languages and modalities. The lab aims to promote the development of robust, transparent, and globally inclusive financial AI systems, with datasets and evaluation resources publicly released to support reproducible research.

💡 Research Summary

**
The paper introduces the CLEF‑2026 FinMMEval Lab, the first evaluation framework that simultaneously addresses multilingual and multimodal challenges for financial large language models (LLMs). Recognizing that existing financial NLP benchmarks are predominantly English‑only and text‑only, the authors design a three‑task suite that progresses from knowledge acquisition to analytical integration and finally to action‑oriented decision making.

Task 1 – Financial Exam Question Answering provides professional‑level multiple‑choice questions drawn from certifications such as CFA, EFA, CPA, and region‑specific exams. The dataset spans five to seven languages (English, Chinese, Arabic, Hindi, Greek, Japanese, Spanish) with 200–600 items per language. Models must select the correct option; performance is measured by accuracy. Participants may evaluate on any subset of languages, enabling both monolingual and multilingual configurations.

Task 2 – Multilingual Financial Question Answering (PolyFiQA) combines U.S. SEC 10‑K/10‑Q excerpts (English) with multilingual news articles (English, Chinese, Japanese, Spanish, Greek) about the same company. Two difficulty tiers are defined: “Easy” (factual or numeric trend queries) and “Expert” (complex analytical questions requiring multi‑document reasoning). The benchmark contains 344 QA instances, each with a concise (≤ 100‑word) evidence‑grounded answer. Primary evaluation uses ROUGE‑1; BLEURT and factual consistency scores serve as secondary metrics, capturing both linguistic quality and truthfulness.

Task 3 – Financial Decision Making moves beyond static QA to a “reasoning‑to‑action” scenario. Daily market contexts are provided for two assets: Bitcoin (crypto) and Tesla (equity). Each day’s JSON record includes a single price point, a synthesized news summary, a manually annotated momentum label (bullish/neutral/bearish), and, for TSLA, fundamental filings (10‑K/10‑Q). Models must output a discrete action (Buy, Hold, Sell) together with a brief rationale (≤ 50 words) that cites supporting evidence. Evaluation focuses on profitability and risk management: cumulative return is the primary metric, complemented by Sharpe Ratio, maximum drawdown, daily volatility, and annualized volatility.

All datasets were curated and validated by financial professionals and native speakers, achieving inter‑annotator agreement above 89 %. The resources are released under an MIT license, ensuring reproducibility and encouraging community contributions.

The authors position FinMMEval as a significant step forward because it (1) expands language coverage beyond English, (2) integrates textual, tabular, and numeric modalities, and (3) establishes a hierarchical evaluation pipeline that mirrors real‑world financial expertise—from exam‑style knowledge checks, through multilingual analytical QA, to actionable trading decisions.

Limitations are acknowledged: the current language set is still limited, and visual modalities such as charts or graphs are absent. Future editions aim to incorporate low‑resource languages, additional multimodal signals (e.g., financial charts, regulatory filing PDFs), and real‑time streaming evaluation to further close the gap between research benchmarks and production‑grade financial AI systems.

Overall, the FinMMEval Lab provides a comprehensive, transparent, and publicly available platform for assessing the robustness, generalization, and practical utility of multilingual, multimodal financial LLMs, thereby fostering the development of globally inclusive and trustworthy financial AI.