用强化学习训练小模型在动态迷宫中自主导航并复用计算
HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning
- 基于强化学习训练递归推理模型，实现动态环境下的持续决策
- 在不确定迷宫中成功导航，且推理过程能复用历史计算
- 适合研究小型智能体在复杂现实场景中的自适应能力
层级推理模型（HRM）虽规模小却具备出色推理能力，但此前仅用于监督式、静态、完全可观测问题。其优势在于可根据问题难度调整计算量，但在动态、不确定或部分可观测环境中无法复用先前时间步的计算，也无法处理正确动作未定义的问题，而这正是许多真实场景的特征。本文提出HRM-Agent，一种仅通过强化学习训练的HRM变体。实验表明，该模型能在动态且不确定的迷宫环境中学会导航。近期研究认为HRM的推理能力源于其递归推理过程。我们分析了该过程的动态特性，发现其能够有效复用早期环境时间步的计算，证明其具备持续学习与状态记忆能力。
原文摘要 · Abstract (English)
The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。