用塔防游戏测试大模型决策与规划能力,发现明显差距。
TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents
- 设计轻量级多模态塔防环境,支持文本+图像+结构化数据输入。
- 五种难度级别评测显示大模型表现远逊人类专家,且易产生幻觉。
- 适合研究大模型代理行为、幻觉检测及策略生成的科研人员使用。
大语言模型(LLMs)在长期规划与决策方面展现出成为通用智能体的潜力。实时战略(RTS)游戏是评估此类能力的理想场景,因其需兼顾宏观策略与微观执行。现有环境或计算成本高,或不支持文本观察,限制了对大模型的评估。为此,我们提出TowerMind——一个基于塔防(TD)类别的新型环境。它兼具低计算开销与多模态观测空间(像素、文本、结构化状态),支持模型幻觉评估并高度可定制。我们设计了五个基准关卡,在不同多模态输入下评测多个主流大模型。结果表明,大模型在能力与幻觉控制上均显著落后于人类专家。实验还揭示其行为缺陷:缺乏计划验证、决策缺乏多样性、动作使用效率低。此外,我们对比了经典强化学习算法Ape-X DQN与PPO。TowerMind以轻量化、多模态设计,填补了现有环境空白,为人工智能代理研究提供新基准。代码已开源于GitHub(https://github.com/tb6147877/TowerMind)。
原文摘要 · Abstract (English)
Recent breakthroughs in Large Language Models (LLMs) have positioned them as a promising paradigm for agents, with long-term planning and decision-making emerging as core general-purpose capabilities for adapting to diverse scenarios and tasks. Real-time strategy (RTS) games serve as an ideal testbed for evaluating these two capabilities, as their inherent gameplay requires both macro-level strategic planning and micro-level tactical adaptation and action execution. Existing RTS game-based environments either suffer from relatively high computational demands or lack support for textual observations, which has constrained the use of RTS games for LLM evaluation. Motivated by this, we present TowerMind, a novel environment grounded in the tower defense (TD) subgenre of RTS games. TowerMind preserves the key evaluation strengths of RTS games for assessing LLMs, while featuring low computational demands and a multimodal observation space, including pixel-based, textual, and structured game-state representations. In addition, TowerMind supports the evaluation of model hallucination and provides a high degree of customizability. We design five benchmark levels to evaluate several widely used LLMs under different multimodal input settings. The results reveal a clear performance gap between LLMs and human experts across both capability and hallucination dimensions. The experiments further highlight key limitations in LLM behavior, such as inadequate planning validation, a lack of multifinality in decision-making, and inefficient action use. We also evaluate two classic reinforcement learning algorithms: Ape-X DQN and PPO. By offering a lightweight and multimodal design, TowerMind complements the existing RTS game-based environment landscape and introduces a new benchmark for the AI agent field. The source code is publicly available on GitHub(https://github.com/tb6147877/TowerMind).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。