评测大模型在真实金融场景中调用工具的能力，构建了包含613个样本的基准测试集。
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
- 基于真实金融模型上下文协议设计多层级任务场景
- 涵盖65个真实金融MCP和三类复杂度任务，覆盖单工具、多工具与多轮交互
- 适用于评估金融领域大模型代理的工具调用精度与推理能力
本文提出 extbf{FinMCP-Bench}，一个用于评估大语言模型（LLMs）通过调用金融模型上下文协议（MCP）解决真实金融问题的新基准。该基准包含613个样本，覆盖10个主场景和33个子场景，包含真实与合成用户查询以确保多样性与真实性。它整合了65个真实金融MCP，并设计了三类样本：单工具、多工具和多轮交互，支持在不同任务复杂度下评估模型表现。利用该基准，我们系统评估了多种主流大模型，并提出了明确衡量工具调用准确率与推理能力的指标。FinMCP-Bench为金融领域大模型代理研究提供了标准化、实用且具有挑战性的测试平台。
原文摘要 · Abstract (English)
This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。