测试大模型在真实财报中的财务推理能力，发现其依赖表面模式而非真正逻辑。
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
- 构建超长财报数据集FinIndices，含未裁剪的多期财务表与对抗陷阱。
- 模型在无公式提示时性能暴跌，暴露时间推导和会计匹配缺陷。
- 结构复杂度高时模型退化为简单取数或凑算，适合金融可信推理研究者。
大型语言模型（LLMs）是否具备真实的结构化推理能力，还是仅依赖表面模式匹配？金融领域因需数值精确和多步逻辑推理，且上下文跨度长，是理想的测试场景。现有基准无法反映真实工业复杂性，主要依赖多选题或截断表格的单跳问答，忽视了跨报表动态和时间去累积机制。为此，我们提出FinIndices，一个大规模基准，评估模型在未裁剪财务报表（最长32K token）上的数据处理保真度。通过自动化合成管道与对抗陷阱，涵盖单指标计算和表级指标汇总，以检验复杂领域、时间序列及计量逻辑推理能力。评估揭示两个严重漏洞：首先，“知识瓶颈”——尽管预训练中记住了公式，但模型表现出脆弱的模式匹配；移除显式公式提示后，性能骤降（如Gemini-3.1-Pro在表任务上从70.70%降至38.22%），暴露出时间去累积与存量流量计量错配的致命缺陷。其次，“结构瓶颈”——生成多指标、多周期表格带来的认知负荷会耗尽推理能力，导致模型从精准推导退化为浅层启发式策略，如错误获取相邻列或用简单算术替代深层会计调整。最后，监督微调（SFT）带来显著零提示提升（单指标+8.54%，表任务+3.82%），验证通过数据对齐可部分恢复结构化逻辑。
原文摘要 · Abstract (English)
Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K tokens). Utilizing an automated synthesis pipeline with adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。