首个针对个性化工具的LLM代理评测基准，聚焦真实场景挑战。
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
- 构建模拟个人应用环境的评测框架，涵盖社交与协作平台。
- SOTA代理在个性化任务中表现不佳，暴露实际应用短板。
- 适合研究智能体泛化能力及个人化工具集成的开发者使用。
模型上下文协议（MCP）已成为连接大语言模型（LLMs）与外部数据源和工具的变革性标准，广泛应用于个人应用与开发平台。然而，现有评测主要关注通用信息查询工具，未能反映个人社交应用中的实际挑战——即工具需与个人账号或本地数据库交互。为此，我们提出MCP-Persona，首个专为评估代理在真实世界个性化MCP工具上表现而设计的基准。该基准涵盖广泛使用的应用，包括社交平台如Reddit和Xiaohongshu（Rednote），以及企业协作套件如Lark（Feishu）和Slack。我们在多种先进代理上进行广泛实验，发现其在个性化工具使用中存在显著困难，凸显该基准在识别并解决此类局限性中的关键作用。MCP-Persona已公开于https://github.com/wwh0411/MCP-Persona。
原文摘要 · Abstract (English)
The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and tools, and has been rapidly adopted across personal applications and development platforms. However, existing benchmarks predominantly focus on generic information-seeking tools and fail to capture the practical challenges posed by personal social applications, where tools interact with individual accounts or local databases. To bridge this critical gap, we introduce MCP-Persona, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools. MCP-Persona encompasses a diverse set of widely-used applications, ranging from social media platforms like Reddit and Xiaohongshu (Rednote) to enterprise collaboration suites such as Lark (Feishu) and Slack. Our extensive experiments on various state-of-the-art (SOTA) agents demonstrate their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations. MCP-Persona is publicly available at https://github.com/wwh0411/MCP-Persona}{https://github.com/wwh0411/MCP-Persona.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。