XFinBench评估大模型在复杂金融问题上的推理能力，发现其仍远低于人类专家。
XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning
- 构建4235个跨金融领域的多模态题库，涵盖术语理解、时间推理等五项核心能力。
- 最优模型o1准确率67.3%，但与人类专家相差12.5%，尤其在时间与情景推理上短板明显。
- 小模型仅在引入3032个金融术语知识库时性能提升，且存在计算舍入与图像位置盲区问题。
解决金融问题需要复杂的推理能力、多模态数据处理和广泛的技术知识，对当前大语言模型构成独特挑战。我们提出XFinBench，一个包含4,235个示例的新基准，用于评估大模型在多样化研究生级金融主题中解决知识密集型复杂问题的能力，支持多模态上下文。通过XFinBench，我们识别出大模型的五大核心能力：术语理解、时间推理、未来预测、情景规划和数值建模。我们在18个领先模型上进行广泛实验，结果显示：o1是表现最佳的纯文本模型，整体准确率为67.3%，但仍显著落后于人类专家（相差12.5%），尤其在时间推理和情景规划方面。我们进一步构建了含3,032个金融术语的知识库，用于知识增强分析，发现相关知识仅能持续提升小型开源模型的性能。此外，错误分析揭示：计算中的舍入误差，以及对图像中曲线位置与交点的忽视，是导致模型在计算题和视觉上下文题上表现不佳的两大主因。代码与数据集已开源：https://github.com/Zhihan72/XFinBench。
原文摘要 · Abstract (English)
Solving financial problems demands complex reasoning, multimodal data processing, and a broad technical understanding, presenting unique challenges for current large language models (LLMs). We introduce XFinBench, a novel benchmark with 4,235 examples designed to evaluate LLM's ability in solving complex, knowledge-intensive financial problems across diverse graduate-level finance topics with multi-modal context. We identify five core capabilities of LLMs using XFinBench, i.e, terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling. Upon XFinBench, we conduct extensive experiments on 18 leading models. The result shows that o1 is the best-performing text-only model with an overall accuracy of 67.3%, but still lags significantly behind human experts with 12.5%, especially in temporal reasoning and scenario planning capabilities. We further construct a knowledge bank with 3,032 finance terms for knowledge augmentation analysis, and find that relevant knowledge to the question only brings consistent accuracy improvements to small open-source model. Additionally, our error analysis reveals that rounding errors during calculation and blindness to position and intersection of curves in the image are two primary issues leading to model's poor performance in calculating and visual-context questions, respectively. Code and dataset are accessible via GitHub: https://github.com/Zhihan72/XFinBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。