构建可验证的金融推理基准，自动生成无误差题目测试大模型
V-FiLLM: Verified Financial LLM Reasoning Benchmark
- 用可执行计算树生成金融问题，答案正确性由构造保证
- 深度增加导致准确率下降51%，对抗扰动下下降47个百分点
- 轻量微调可提升准确率，适合金融问答任务优化
现有基准在科学、技术、工程和数学领域已取得显著进展，但对结构化数据上的金融推理研究仍较薄弱。我们提出V-FiLLM框架，基于真实表格生成金融推理题，通过可执行计算树确保答案正确性。利用符号计算获得真实答案，并转换为自然语言问题，完全脱离模型标注，可无成本大规模生成且不继承生成器错误。该框架包含四个独立可控难度维度：计算深度、表达广度、金融概念复杂度和上下文大小。在开源模型上评估发现，推理深度增加时准确率最高下降51%，对抗性数值扰动下下降47个百分点，凸显当前模型在表格金融推理中的脆弱性。进一步实验表明，在经过验证的思维链轨迹上进行轻量级LoRA微调，可使未见问题准确率从81.1%提升至85.6%，在FinQA数据集上比基线模型高5个百分点，表明针对性低代价适配是提升复合金融问答能力的有效路径。
原文摘要 · Abstract (English)
While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。