Accounting Reasoning in Large Language Models: Concepts, Evaluation, and Empirical Analysis

Large language models (LLMs) are increasingly reshaping learning paradigms, cognitive processes, and research methodologies across diverse domains. As their adoption expands, effectively integrating LLMs into professional fields and clarifying their role in domain-specific applications has become a key challenge for enterprise digital transformation and broader societal development. In the accounting domain, successful integration requires a systematic understanding of LLMs’ domain-specific reasoning capabilities. In this study, we introduce the concept of accounting reasoning and propose a set of evaluation criteria grounded in an analysis of the training data characteristics of representative GLM-series models. These criteria establish a foundation for studying accounting-oriented reasoning paradigms and provide benchmarks for assessing and improving model performance. Building on this framework, we evaluate several representative LLMs, including GLM-6B, GLM-130B, GLM-4, and GPT-4, across a range of accounting reasoning tasks. Our experimental results show that prompt engineering strategies can yield varying degrees of performance improvement across models, with GPT-4 demonstrating the strongest overall accounting reasoning capability. Nevertheless, the results indicate that current LLMs remain insufficient for real-world accounting applications. In particular, further optimization is required for deployment in enterprise-level accounting scenarios to fully realize the potential value of LLMs in this domain.

💡 Research Summary

The paper tackles the emerging need to understand how large language models (LLMs) can be integrated into professional accounting work. It begins by formally defining “accounting reasoning” as the combination of mathematical reasoning, logical inference, and domain‑specific rule application. To evaluate this composite capability, the authors propose three quantitative criteria: (1) reasoning accuracy – does the final answer satisfy both numerical calculations and accounting standards; (2) reasoning consistency – are intermediate steps logically coherent and free from cumulative error; and (3) error‑propagation behavior – how much does an early mistake amplify across multi‑step tasks.

A benchmark dataset is constructed specifically for accounting. Existing math‑reasoning corpora such as GSM8K and its filtered variant MR‑GSM8K are adapted to accounting contexts, and additional items are drawn from Chinese CPA examinations, financial statements, cost‑accounting scenarios, and audit judgment questions. The final suite contains roughly 4,200 items covering four sub‑domains (financial statements, cost accounting, auditing, tax). Each item is structured into a prompt, a sequence of required intermediate reasoning steps, and a ground‑truth answer, enabling fine‑grained analysis of model behavior.

Four representative LLMs are evaluated: GLM‑6B, GLM‑130B, GLM‑4, and GPT‑4. Each model is tested under three prompting regimes – zero‑shot, few‑shot, and chain‑of‑thought (CoT) – to assess the impact of prompt engineering. Results show a clear hierarchy: GPT‑4 consistently outperforms the GLM series, achieving an overall accuracy of 78 % under CoT prompting, especially on tasks that blend multi‑step calculations with rule‑based decisions. GLM‑130B, despite its larger parameter count, lags behind in consistency scores, while GLM‑6B and GLM‑4 remain below 30 % accuracy across the board. Prompt engineering effects are not uniform; CoT boosts performance on mathematically intensive problems but can degrade results on pure rule‑application tasks where a simpler few‑shot prompt yields more stable outputs.

The authors identify several limitations. The benchmark is rooted in Chinese accounting standards, raising questions about generalizability to IFRS or US GAAP. Models are not forced to emit explicit intermediate reasoning traces, which hampers auditability and alignment with real‑world accounting workflows. Moreover, current LLMs lack explanatory “why” statements, limiting their usefulness for audit trails and regulatory compliance.

To address these gaps, the paper proposes future research directions: (1) domain‑specific pre‑training or fine‑tuning on large corpora of accounting standards and regulatory texts; (2) hybrid architectures that combine LLMs with external symbolic engines or rule‑based modules to enforce hard constraints and reduce error propagation; and (3) systematic logging frameworks that capture step‑by‑step reasoning for downstream verification.

In conclusion, the study provides a rigorous conceptualization of accounting reasoning, a novel benchmark tailored to the field, and an empirical comparison that highlights GPT‑4’s relative strength while underscoring the insufficiency of existing models for production‑grade accounting tasks. The work establishes a foundation for future efforts to build trustworthy, reasoning‑capable AI assistants for the accounting profession.