用游戏分解法透明评估大模型推理能力，避免数据污染。
GAMEBoT: Transparent Assessment of LLM Reasoning in Games
- 将游戏复杂推理拆解为模块化子问题，引导模型逐步思考。
- 17个主流大模型在8个动态游戏中表现普遍不理想，即使有详细提示。
- 适合关注模型真实推理能力、需防数据污染的研究者使用。
大型语言模型（LLMs）在需要复杂推理的现实应用中日益普及。为有效追踪进展，亟需能超越表面模式识别的可靠评估基准。然而，现有推理评测常面临可解释性不足、性能饱和或数据污染等问题。为此，我们提出GAMEBoT——一个用于严格且透明评估LLM推理能力的游戏竞技场。GAMEBoT将游戏中的复杂推理分解为预定义的模块化子问题，设计一系列基于领域知识的思维链（CoT）提示，引导模型在行动前解决这些子问题。同时，开发规则算法生成子问题的真值，实现对中间推理步骤的严谨验证。该方法可同时评估最终动作质量与推理过程准确性。此外，通过动态游戏和模型间对抗竞赛，自然缓解了数据污染风险。我们在8种不同策略与特性游戏中，对17个主流大模型进行了评测。结果表明，即使提供详细CoT提示，GAMEBoT仍构成显著挑战。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in real-world applications that demand complex reasoning. To track progress, robust benchmarks are required to evaluate their capabilities beyond superficial pattern recognition. However, current LLM reasoning benchmarks often face challenges such as insufficient interpretability, performance saturation or data contamination. To address these challenges, we introduce GAMEBoT, a gaming arena designed for rigorous and transparent assessment of LLM reasoning capabilities. GAMEBoT decomposes complex reasoning in games into predefined modular subproblems. This decomposition allows us to design a suite of Chain-of-Thought (CoT) prompts that leverage domain knowledge to guide LLMs in addressing these subproblems before action selection. Furthermore, we develop a suite of rule-based algorithms to generate ground truth for these subproblems, enabling rigorous validation of the LLMs' intermediate reasoning steps. This approach facilitates evaluation of both the quality of final actions and the accuracy of the underlying reasoning process. GAMEBoT also naturally alleviates the risk of data contamination through dynamic games and head-to-head LLM competitions. We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics. Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts. Project page: https://visual-ai.github.io/gamebot
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。