让大模型通过玩游戏学会动态决策,比传统强化学习更省数据、更透明。
Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models
- 将强化学习转化为语言建模任务,用自然语言生成策略并迭代优化。
- 在游戏环境中表现媲美传统方法,但所需数据和算力大幅降低。
- 每步决策附带中文解释,适合需要可解释性的交互系统场景。
大语言模型在数学、编程等复杂推理任务上表现出色,但在儿童轻易完成的简单互动任务中却常失败。这暴露出陈述性知识(知道什么)与程序性知识(知道如何做)之间的巨大差距。传统强化学习虽能通过环境交互获取程序性知识,但往往为黑箱且需大量训练数据。而大模型虽具备丰富世界知识和推理能力,却难以将其转化为交互场景中的动态决策。为此,我们提出「思考游戏」(Think in Games, TiG)框架,使大模型通过直接与游戏环境互动,发展出程序性理解,同时保留其内在推理与解释能力。具体而言,TiG将基于强化学习的决策重构为语言建模任务:大模型生成语言引导的策略,并根据环境反馈在线迭代优化。实验表明,TiG成功弥合了陈述性与程序性知识的鸿沟,在性能上可媲美传统强化学习方法,但数据和计算需求显著降低。此外,TiG能为每一步决策提供逐步的自然语言解释,极大提升了复杂交互任务中的透明度与可解释性。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at complex reasoning tasks such as mathematics and coding, yet they frequently struggle with simple interactive tasks that young children perform effortlessly. This discrepancy highlights a critical gap between declarative knowledge (knowing about something) and procedural knowledge (knowing how to do something). Although traditional reinforcement learning (RL) agents can acquire procedural knowledge through environmental interaction, they often operate as black boxes and require substantial training data. In contrast, LLMs possess extensive world knowledge and reasoning capabilities, but are unable to effectively convert this static knowledge into dynamic decision-making in interactive settings. To address this challenge, we propose Think in Games (TiG), a novel framework that empowers LLMs to develop procedural understanding through direct interaction with game environments, while retaining their inherent reasoning and explanatory abilities. Specifically, TiG reformulates RL-based decision-making as a language modeling task: LLMs generate language-guided policies, which are refined iteratively through online reinforcement learning based on environmental feedback. Our experimental results show that TiG successfully bridges the gap between declarative and procedural knowledge, achieving competitive performance with dramatically lower data and computational demands compared to conventional RL methods. Moreover, TiG provides step-by-step natural language explanations for its decisions, greatly improving transparency and interpretability in complex interactive tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。