首个俄语金融推理基准，支持步骤可验证的链式思考
RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluation
- 基于可执行Python模板生成5280个参数化金融问题，每题含标准推理路径
- 模型最终答案正确率仅29%，中间步骤对齐率约65%，暴露出显著推理差距
- 引入模糊对齐与软注意力评估，诊断能力优于传统方法，适合俄语金融AI研究者
多步符号推理对稳健金融分析至关重要，但现有基准普遍忽略中间推理过程。FinChain虽引入可验证链式思考（CoT）评估，但仅限英文；FINESSE-Bench虽包含俄语部分，却依赖选择题且无步骤监督。本文提出RusFinChain，首个面向俄语金融领域的可验证链式思考基准，覆盖17个领域、172个主题，包含5,280个由可执行Python模板生成的参数化示例，确保评估无污染。每个示例配有标准推理链及中间数值，支持自动验证。我们引入增强型指标：模糊数值对齐与软注意力对齐。在分层样本上评估8个开源大模型，生成8,100条响应。结果揭示显著推理差距：模型步骤对齐的Hard F1约为0.65，但最终答案正确率仅为约29%。模糊与软性指标与最终答案正确性相关性更强（斯皮尔曼ρ≈0.48），优于原始ChainEval（ρ≈0.38-0.46），证明其更强诊断能力。数据集、代码与评估框架已开源，推动俄语社区可验证金融AI发展。
原文摘要 · Abstract (English)
Multi-step symbolic reasoning is essential for robust financial analysis, yet most benchmarks neglect intermediate reasoning steps. FinChain introduced verifiable Chain-of-Thought (CoT) evaluation but is limited to English. FINESSE-Bench includes a Russian block but relies on multiple-choice questions without step-level supervision. We present RusFinChain, the first Russian-language symbolic benchmark for verifiable CoT reasoning in finance. It spans 17 domains, 172 topics, and comprises 5,280 parameterized examples from executable Python templates, ensuring contamination-free evaluation. Each example includes a gold-standard reasoning chain with intermediate numeric values for automatic verification. We also introduce enhanced metrics: Fuzzy Numeric Alignment and Soft-Attention Alignment. We evaluate 8 open-weight LLMs on a stratified sample, generating 8,100 responses. Results reveal a substantial reasoning gap: models achieve Hard F1 of ~0.65 for step alignment, but only ~29% of final answers are correct. Our fuzzy and soft metrics show stronger correlation with final-answer correctness (Spearman rho approx 0.48) than the original ChainEval (rho approx 0.38-0.46), demonstrating superior diagnostic power. We release dataset, code, and evaluation framework to foster verifiable financial AI for the Russian-speaking community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。