arXiv:2506.04098cs.CLcs.AI2025-06被引 4

用文字描述游戏状态,测试语言模型在10万步长任务中的决策能力。

TextAtari: 100K Frames Game Playing with Language Agents

  • 将经典雅达利游戏转为文本描述,构建长时序决策评测基准
  • 三类大模型在10万步任务中表现远低于人类玩家
  • 适合研究语言模型与长期规划结合的学者参考

我们提出TextAtari,一个用于评估语言代理在长达10万步的长时序决策任务中的基准。通过将经典雅达利游戏的视觉状态表示转化为丰富的文本描述,TextAtari构建了一个连接序列决策与自然语言处理的挑战性测试平台。该基准包含近100个不同复杂度、动作空间和规划时长的任务,全部通过无监督表征学习框架(AtariARI)转为文本。我们在三种代理框架(零样本、少量样本思维链、反思推理)下评估了三个开源大语言模型(Qwen2.5-7B、Gemma-7B、Llama3.1-8B),探究先验知识对长时序挑战的影响。四个场景(基础、模糊、手动增强、基于参考)研究了语义理解、指令解析与专家示范对代理决策的作用。结果揭示语言代理在大规模规划任务中与人类玩家存在显著差距,暴露出序列推理、状态追踪与战略规划等方面的挑战。TextAtari提供标准化评估协议、基线实现及研究框架,推动语言模型与规划交叉领域发展。代码已公开于https://github.com/Lww007/Text-Atari-Agents。

原文摘要 · Abstract (English)

We present TextAtari, a benchmark for evaluating language agents on very long-horizon decision-making tasks spanning up to 100,000 steps. By translating the visual state representations of classic Atari games into rich textual descriptions, TextAtari creates a challenging test bed that bridges sequential decision-making with natural language processing. The benchmark includes nearly 100 distinct tasks with varying complexity, action spaces, and planning horizons, all rendered as text through an unsupervised representation learning framework (AtariARI). We evaluate three open-source large language models (Qwen2.5-7B, Gemma-7B, and Llama3.1-8B) across three agent frameworks (zero-shot, few-shot chain-of-thought, and reflection reasoning) to assess how different forms of prior knowledge affect performance on these long-horizon challenges. Four scenarios-Basic, Obscured, Manual Augmentation, and Reference-based-investigate the impact of semantic understanding, instruction comprehension, and expert demonstrations on agent decision-making. Our results reveal significant performance gaps between language agents and human players in extensive planning tasks, highlighting challenges in sequential reasoning, state tracking, and strategic planning across tens of thousands of steps. TextAtari provides standardized evaluation protocols, baseline implementations, and a framework for advancing research at the intersection of language models and planning. Our code is available at https://github.com/Lww007/Text-Atari-Agents.

语言模型长时序决策强化学习文本生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。