arXiv:2506.02515cs.CLcs.AI2025-06ACL被引 18

首个可验证金融推理链的基准,专为透明化多步财务分析设计。

FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning

  • 基于符号模板与可执行代码生成可验证的金融推理链条。
  • 26个主流大模型在多步推理上表现不佳,即使顶尖模型也存明显短板。
  • 适合研究可信、可解释金融AI的学者和开发者使用。

多步符号推理对稳健的金融分析至关重要,但现有基准大多忽略这一能力。当前数据集如FinQA和ConvFinQA侧重最终数值答案,忽视了透明度和可验证性所需的中间推理步骤。为弥补这一空白,我们提出FinChain,首个专为金融领域可验证推理链设计的基准。FinChain覆盖12个金融领域中的58个主题,每个主题由带可执行Python代码的参数化符号模板表示,支持完全机器可验证的推理和可扩展、无污染的数据生成。为评估推理能力,我们提出CHAINEVAL,一种动态对齐度量,联合评估最终答案正确性与步骤级推理一致性。对26个领先大模型的评估显示,即使前沿模型在符号金融推理上仍有明显局限;而经过领域适配和数学增强微调的模型能显著缩小差距。总体而言,FinChain揭示了多步金融推理中的持续弱点,并为开发可信、可解释、可验证的金融AI奠定了基础。项目地址:https://github.com/mbzuai-nlp/finchain.git。

原文摘要 · Abstract (English)

Multi-step symbolic reasoning is essential for robust financial analysis; yet, current benchmarks largely overlook this capability. Existing datasets such as FinQA and ConvFinQA emphasize final numerical answers while neglecting the intermediate reasoning steps required for transparency and verification. To address this gap, we introduce FinChain, the first benchmark specifically designed for verifiable Chain-of-Thought evaluation in finance. FinChain spans 58 topics across 12 financial domains, each represented by parameterized symbolic templates with executable Python code that enable fully machine-verifiable reasoning and scalable, contamination-free data generation. To assess reasoning capacity, we propose CHAINEVAL, a dynamic alignment measure that jointly evaluates both the final-answer correctness and the step-level reasoning consistency. Our evaluation of 26 leading LLMs reveals that even frontier LLMs exhibit clear limitations in symbolic financial reasoning, while domain-adapted and math-enhanced fine-tuned models can substantially narrow this gap. Overall, FinChain exposes persistent weaknesses in multi-step financial reasoning and provides a foundation for developing trustworthy, interpretable, and verifiable financial AI. This project is available at https://github.com/mbzuai-nlp/finchain.git.

金融AI推理链可验证性大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。