测试大模型在限时策略游戏中表现，发现系统级能力比纯推理更重要。
Evaluating Large Language Models as Live Strategic Agents: Provider Performance, Hybrid Decomposition, and Operational Gaps in Timed Risk Play
- 构建限时多阶段风险游戏环境，模拟真实部署中的反复规划与执行。
- Gemini在32局比赛中赢20局，但拆分后规划能力与其他模型接近均等。
- 胜败关键在于目标追踪、执行转化率和运行稳定性，适合系统设计者参考。
静态基准仅反映大语言模型的部分实际行为。真实系统中，模型需在时间限制、格式约束和失败模式下进行重复循环的规划与执行。我们在此类场景下，于一个具有明确胜利目标的限时多阶段风险游戏中展开研究。在规则冻结的32局跨提供商锦标赛中，gemini-3.1-pro-preview击败gpt-5.1、claude-opus-4-7和kimi-k2.6，赢得20局，整体胜率分布显著偏离均等假设（p ≈ 1.5 × 10^-5）。随后，通过将执行环节标准化为更低成本的Gemini Flash框架，对规划能力独立评估。此时32局规划对比结果与近似均等一致（p ≈ 0.821），表明早期性能差异主要来自端到端系统行为，而非仅规划能力。进一步分析保存的规划与执行轨迹发现，Gemini更频繁地关注终局目标，且在接近胜利时强化此聚焦；同时，其将更多回合转化为深层征服链，尽管运行表现并非最稳定。结果表明，实时代理性能取决于目标追踪、执行转化、成本控制与运行可靠性，支持将大模型作为受限工作流中的组件来评估，而非孤立的基准响应体。
原文摘要 · Abstract (English)
Static benchmarks capture only part of how large language models behave in practice. Real systems place models inside repeated loops with time limits, formatting constraints, and failure modes. We study this setting in a timed multi-phase Risk environment with explicit victory targets and repeated planning and execution cycles. In a replicated 32-game cross-provider championship under frozen rules, gemini-3.1-pro-preview won 20 of 32 games against gpt-5.1, claude-opus-4-7, and kimi-k2.6, and the pooled winner distribution differs strongly from an equal-strength null (p approx 1.5 x 10^-5). We then separate planning from execution by standardizing execution on a cheaper Gemini Flash scaffold. Under this design, a pooled 32-game planner bakeoff is consistent with near-equality (p approx 0.821), which indicates that much of the earlier provider spread came from end-to-end system behavior rather than planning alone. To study mechanism, we analyze saved planning and execution traces from the provider championship. Gemini refers to the terminal objective far more often than the other models and increases that focus as victory approaches. Gemini also converts more turns into deep conquest chains, even though it is not the cleanest runtime. These results show that live-agent performance depends on objective tracking, execution conversion, cost, and runtime reliability, and they support evaluating LLMs as components in bounded workflows rather than as isolated benchmark respondents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。