用游戏测试大模型的动态规则推理能力，发现仍存根本性困难
Baba is LLM: Reasoning in a Game with Dynamic Rules
- 让大模型玩可改规则的文本解谜游戏，考察语言与推理结合能力
- 大模型（如GPT-4o）表现较好，但小模型难以理解游戏机制
- 微调提升分析能力，但解决策略生成仍弱，适合研究推理瓶颈
大型语言模型（LLMs）在语言任务中表现优异，但在推理任务中表现不佳。本文探讨了LLMs在2D解谜游戏Baba is You中的表现，玩家需通过移动定义物体属性的文本块来修改游戏规则。由于规则操作依赖语言理解和逻辑推理，该任务对LLMs是有力挑战。评估了六种不同提示方式（简单、规则扩展、动作扩展）下的模型表现，并对Mistral和OLMo两个模型使用游戏中的文本与结构数据进行微调。结果显示，较大模型（特别是GPT-4o）在推理和解谜中表现更优，而较小的未微调模型难以识别游戏机制或应用规则变化。微调提升了对关卡的分析能力，但未显著改善解法生成。结论表明，即使是最先进的微调后的模型，在处理动态规则变更时仍存在困难（尤其在使用-提及区分上）。研究为评估大模型在复杂任务中的适用性提供了洞见，并强调具有动态规则的游戏适合作为测试大模型推理与反思能力的基准。
原文摘要 · Abstract (English)
Large language models (LLMs) are known to perform well on language tasks, but struggle with reasoning tasks. This paper explores the ability of LLMs to play the 2D puzzle game Baba is You, in which players manipulate rules by rearranging text blocks that define object properties. Given that this rule-manipulation relies on language abilities and reasoning, it is a compelling challenge for LLMs. Six LLMs are evaluated using different prompt types, including (1) simple, (2) rule-extended and (3) action-extended prompts. In addition, two models (Mistral, OLMo) are finetuned using textual and structural data from the game. Results show that while larger models (particularly GPT-4o) perform better in reasoning and puzzle solving, smaller unadapted models struggle to recognize game mechanics or apply rule changes. Finetuning improves the ability to analyze the game levels, but does not significantly improve solution formulation. We conclude that even for state-of-the-art and finetuned LLMs, reasoning about dynamic rule changes is difficult (specifically, understanding the use-mention distinction). The results provide insights into the applicability of LLMs to complex problem-solving tasks and highlight the suitability of games with dynamically changing rules for testing reasoning and reflection by LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。