Beyond Knowledge to Agency: Evaluating Expertise, Autonomy, and Integrity in Finance with CNFinBench

As large language models (LLMs) become high-privilege agents in risk-sensitive settings, they introduce systemic threats beyond hallucination, where minor compliance errors can cause critical data leaks. However, existing benchmarks focus on rule-based QA, lacking agentic execution modeling, overlooking compliance drift in adversarial interactions, and relying on binary safety metrics that fail to capture behavioral degradation. To bridge these gaps, we present CNFinBench, a comprehensive benchmark spanning 29 subtasks grounded in the triad of expertise, autonomy, and integrity. It assesses domain-specific capabilities through certified regulatory corpora and professional financial tasks, reconstructs end-to-end agent workflows from requirement parsing to tool verification, and simulates multi-turn adversarial attacks that induce behavioral compliance drift. To quantify safety degradation, we introduce the Harmful Instruction Compliance Score (HICS), a multi-dimensional safety metric that integrates risk-type-specific deductions, multi-turn consistency tracking, and severity-adjusted penalty scaling based on fine-grained violation triggers. Evaluations over 22 open-/closed-source models reveal: LLMs perform well in applied tasks yet lack robust rule understanding, suffer a 15.4 decline from single modules to full execution chains, and collapse rapidly in multi-turn attacks, with average violations surging by 172.3% in Round 2. CNFinBench is available at https://cnfinbench.opencompass.org.cn and https://github.com/VertiAIBench/CNFinBench.

💡 Research Summary

The paper addresses a pressing gap in the evaluation of large language models (LLMs) that are increasingly being deployed as high‑privilege agents in finance, a domain where even minor compliance slips can trigger severe data leaks, regulatory penalties, or market disruptions. Existing safety benchmarks focus largely on static question‑answering or binary “safe/unsafe” judgments, which overlook the dynamic, execution‑oriented nature of real‑world financial workflows and the gradual compliance drift that can emerge under adversarial interaction. To fill this void, the authors introduce CNFinBench, a comprehensive benchmark built around three pillars: Expertise, Autonomy, and Integrity.

Expertise evaluates domain knowledge by leveraging certified regulatory corpora (e.g., Basel III, MiFID II, IFRS) and professional financial tasks such as risk‑adjusted portfolio construction, audit report generation, and tax‑optimization calculations. The benchmark measures term‑mapping accuracy, correct citation of regulatory clauses, and numerical error rates, thereby testing whether the model truly “understands” financial law rather than merely regurgitating jargon.

Autonomy models end‑to‑end agentic execution. A task is decomposed into requirement parsing, tool selection (e.g., market‑data APIs, pricing engines), parameter generation, API invocation, result verification, and final report synthesis. The benchmark checks that the model can autonomously handle tool‑call failures, retry logic, and fallback strategies, exposing weaknesses that are invisible in pure QA settings.

Integrity probes the model’s ability to maintain compliance over multi‑turn dialogues, especially under adversarial prompting designed to induce “drift.” The authors craft multi‑turn attack scenarios where a malicious user subtly nudges the model toward illicit advice, confidential data leakage, or insider‑trading suggestions. To quantify the resulting degradation, they propose the Harmful Instruction Compliance Score (HICS), a multi‑dimensional metric that assigns risk‑type‑specific penalties, tracks consistency across turns, and applies severity‑adjusted scaling based on fine‑grained violation triggers. HICS thus captures not only isolated infractions but also cumulative compliance erosion.

The benchmark comprises 29 subtasks spanning pure knowledge checks, tool‑augmented workflows, and adversarial interaction loops. The authors evaluate 22 models—including open‑source LLaMA, Falcon, GPT‑NeoX, and closed‑source GPT‑4, Claude, and others—under both single‑module and full‑pipeline conditions. Key findings are:

1. Strong applied‑task performance but weak rule comprehension – Models achieve >80 % accuracy on applied financial tasks (e.g., portfolio rebalancing) yet only ~62 % accuracy on precise regulatory clause citation.
2. Significant drop when chaining modules – Transitioning from isolated modules to full execution pipelines incurs an average 15.4 % performance decline, primarily due to interface mismatches and missing tool‑call parameters.
3. Rapid compliance collapse under adversarial multi‑turn attacks – In Round 2 of the attack sequences, average violations surge by 172.3 %, and HICS scores plummet, indicating that models quickly lose adherence to compliance constraints once nudged.

These results demonstrate that current LLMs, while competent at “knowledge‑heavy” tasks, lack robust rule understanding, reliable autonomous execution, and resilience against compliance‑drift attacks. CNFinBench, released openly at https://cnfinbench.opencompass.org.cn and https://github.com/VertiAIBench/CNFinBench, provides the community with a reproducible platform to benchmark, diagnose, and improve financial LLM safety.

Future work outlined by the authors includes expanding the benchmark to additional financial sub‑domains such as insurance and derivatives, integrating real‑time market data streams, evaluating human‑AI collaborative workflows, and developing mechanisms for continual regulatory updates. By doing so, CNFinBench aims to evolve from a static test suite into a living governance framework that supports responsible, high‑stakes deployment of AI agents in finance.