用文字冒险游戏测试大模型长期自主推理能力
TextQuests: How Good are LLMs at Text-Based Video Games?
- 基于经典文字游戏构建，强制模型独立解谜
- 单次会话需持续推理，考验长上下文理解力
- 适合评估大模型在无外部工具下的探索能力
在复杂交互环境中评估智能体的实际能力对理解其真实水平至关重要。现有基准多聚焦于工具使用或结构化任务表现，却难以全面反映智能体在需要持续自主推理的探索性环境中的能力。为此，我们提出TextQuests，基于Infocom系列文字冒险游戏构建的基准。这些游戏对人类玩家平均耗时超30小时，需数百次精确操作才能通关，是评估智能体在专注、状态依赖任务中表现的有效代理。该基准通过禁止使用外部工具，仅依赖大模型自身能力，重点考察其在单次交互会话中进行试错学习与持续解题的内在长上下文推理能力。相关资源已公开于https://textquests.ai。
原文摘要 · Abstract (English)
Evaluating AI agents within complex, interactive environments that mirror real-world challenges is critical for understanding their practical capabilities. While existing agent benchmarks effectively assess skills like tool use or performance on structured tasks, they often do not fully capture an agent's ability to operate autonomously in exploratory environments that demand sustained, self-directed reasoning over a long and growing context. To enable a more accurate assessment of AI agents in challenging exploratory environments, we introduce TextQuests, a benchmark based on the Infocom suite of interactive fiction games. These text-based adventures, which can take human players over 30 hours and require hundreds of precise actions to solve, serve as an effective proxy for evaluating AI agents on focused, stateful tasks. The benchmark is specifically designed to assess an LLM agent's capacity for self-contained problem-solving by precluding the use of external tools, thereby focusing on intrinsic long-context reasoning capabilities in an exploratory environment characterized by the need for trial-and-error learning and sustained problem-solving within a single interactive session. We release TextQuests at https://textquests.ai.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。