构建金融领域评估框架，拆解大模型的知识与推理能力。
FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning
- 分离知识与推理评分，独立量化大模型在金融任务中的表现
- 发现顶级模型仍受限于知识应用，高阶认知能力是关键瓶颈
- 开源22个子领域的中文金融推理数据集，支持可复现研究
大型语言模型（LLMs）在复杂金融推理任务中展现出巨大潜力，但面临同时依赖领域知识和高级推理能力的挑战。现有评估基准常将两类能力混同于单一任务表现，且缺乏对失败原因的溯源分析。为此，我们提出FinEval-KR——一个专用于金融领域的评估框架，能够独立解耦并量化模型的知识与推理能力，引入知识得分与推理得分两个指标。受认知科学启发，我们进一步基于布卢姆分类学提出认知得分，用于分析不同认知层级下的推理表现。同时，我们发布一个开源的中文金融推理数据集，覆盖22个子领域，支持可复现研究与进一步发展。实验结果表明，推理能力和高阶认知能力是影响推理准确性的核心因素。我们还发现，即使顶尖模型在知识应用上仍存在明显瓶颈。此外，分析显示，专用金融LLM在多数指标上普遍落后于顶尖通用大模型。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate significant potential but face challenges in complex financial reasoning tasks requiring both domain knowledge and sophisticated reasoning. Current evaluation benchmarks often fall short by not decoupling these capabilities indicators from single task performance and lack root cause analysis for task failure. To address this, we introduce FinEval-KR, a novel evaluation framework for decoupling and quantifying LLMs' knowledge and reasoning abilities independently, proposing distinct knowledge score and reasoning score metrics. Inspired by cognitive science, we further propose a cognitive score based on Bloom's taxonomy to analyze capabilities in reasoning tasks across different cognitive levels. We also release a new open-source Chinese financial reasoning dataset covering 22 subfields to support reproducible research and further advancements in financial reasoning. Our experimental results reveal that LLM reasoning ability and higher-order cognitive ability are the core factors influencing reasoning accuracy. We also specifically find that even top models still face a bottleneck with knowledge application. Furthermore, our analysis shows that specialized financial LLMs generally lag behind the top general large models across multiple metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。