评测大模型在文明6中长期任务下的工具使用与规划能力
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
- 通过工具协议构建300+回合的长周期任务环境
- 发现模型普遍忽视战略状态查询，近程承诺执行率仅48%-66%
- 适合研究大模型决策可靠性与规划一致性的人工智能学者
我们提出CivBench，一个基于模型上下文协议（MCP）的开源基准，用于评估语言模型代理在长周期、工具驱动环境中的表现。单个实验持续300+回合，产生数千次工具调用，覆盖广阔动作空间，要求持续规划、状态监控与部分可观测下的执行。环境提供76个MCP工具及叙事层，将视觉游戏状态转为结构化文本。我们在23次可接受运行中评估了四个模型家族的表现。样本为试点，非模型排名：聚合结果无法可靠区分模型。为此引入两项界面级度量：主动监控率（PMR），衡量代理是否主动查询潜在战略状态；RAG@10，衡量规划反思中的承诺是否在10回合内执行。观察到两个稳定模式：尽管剧本指导每20回合查询胜利进度，实际仅每30至75回合查询一次，在20次可检测失败中，7次未在预警窗口内查询。同时，代理常未能执行自身规划中的近期承诺（RAG@10为48.2%至65.8%）。这些现象虽有工具访问和明确指导，仍表现为指令偏离而非能力缺失。代码、场景、日志、度量与分析管道已开源。
原文摘要 · Abstract (English)
We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode spans 300+ turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale. Instead, we introduce two interface-level metrics that the environment makes measurable: Proactive Monitoring Rate (PMR), capturing whether agents actively query latent strategic state, and RAG@10, capturing whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across runs we observe two consistent patterns under a shared playbook protocol. Agents under-monitor strategically relevant state that is available but requires explicit querying: despite playbook guidance to query victory progress every 20 turns, agents do so only every 30 to 75 turns, and in 7 of 20 detectable defeats they failed to query within the 20 turn warning window before game end. Agents also frequently fail to execute near-term commitments stated in their own planning reflections (RAG@10 between 48.2% and 65.8% across models). Both patterns arise despite tool access and explicit guidance, and we interpret them as deviations under instruction rather than absences of capability. We release the environment, scenarios, logs, metrics, and analysis pipeline at https://github.com/lmwilki/civ6-mcp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。