用文字解谜游戏测试大模型推理能力，发现其在复杂任务上仍逊于人类。
TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoning
- 设计文本解谜游戏基准，评估模型多轮推理与自我修正能力。
- 模型在难题上表现不佳，仅能解决部分中等难度任务。
- 优化推理能力的模型优于指令跟随型，适合研究智能体自洽性。
推理是大语言模型（LLMs）的核心能力，使其能够理解、分析并解决复杂问题。本文提出TextGames，一个专门用于评估LLMs在高要求文本游戏中的表现的基准，这些游戏需要模式识别、空间感知、算术运算和逻辑推理等高级技能。我们分析了模型在单轮与多轮推理中的表现，以及利用反馈进行自我反思以修正后续答案的能力。结果表明，尽管模型在大多数简单和中等难度任务中表现良好，但在更复杂的任务上仍面临显著挑战；相比之下，人类在充分时间内可解决所有任务。此外，通过自我反思，模型在多轮预测中性能有所提升，但依然难以一致地处理序列、计数和复杂规则。值得注意的是，针对推理优化的模型优于侧重指令遵循的预训练模型，凸显了推理能力在解决高度复杂问题中的关键作用。
原文摘要 · Abstract (English)
Reasoning is a fundamental capability of large language models (LLMs), enabling them to comprehend, analyze, and solve complex problems. In this paper, we introduce TextGames, an innovative benchmark specifically crafted to assess LLMs through demanding text-based games that require advanced skills in pattern recognition, spatial awareness, arithmetic, and logical reasoning. Our analysis probes LLMs' performance in both single-turn and multi-turn reasoning, and their abilities in leveraging feedback to correct subsequent answers through self-reflection. Our findings reveal that, although LLMs exhibit proficiency in addressing most easy and medium-level problems, they face significant challenges with more difficult tasks. In contrast, humans are capable of solving all tasks when given sufficient time. Moreover, we observe that LLMs show improved performance in multi-turn predictions through self-reflection, yet they still struggle with sequencing, counting, and following complex rules consistently. Additionally, models optimized for reasoning outperform pre-trained LLMs that prioritize instruction following, highlighting the crucial role of reasoning skills in addressing highly complex problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。