用语言模型帮强化学习机器人从异常状态中恢复，无需依赖复杂判断。
LaMOuR: Leveraging Language Models for Out-of-Distribution Recovery in Reinforcement Learning
- 用语言模型生成恢复指引，引导机器人回到正常状态。
- 在多种运动任务中提升恢复效率，复杂环境表现更优。
- 适合需要鲁棒性的人形机器人与移动操作场景。
深度强化学习在机器人控制中表现优异，但对分布外（OOD）状态敏感，易导致错误动作和任务失败。现有方法多聚焦于避免OOD发生，却忽视了遭遇后的恢复机制。尽管最新研究尝试通过不确定性估计引导回正，但在复杂环境中难以扩展。为此，我们提出语言模型驱动的分布外恢复方法（LaMOuR），无需依赖不确定性估计即可实现恢复学习。LaMOuR利用视觉语言模型（LVLM）的图像描述、逻辑推理与代码生成能力，生成密集奖励编码，指导智能体返回可完成原任务的状态。实验表明，LaMOuR显著提升了多样化运动任务中的恢复效率，且在人形行走与移动操作等复杂场景中具备良好泛化能力，优于现有方法。代码与补充材料见 https://lamour-rl.github.io/。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (DRL) has demonstrated strong performance in robotic control but remains susceptible to out-of-distribution (OOD) states, often resulting in unreliable actions and task failure. While previous methods have focused on minimizing or preventing OOD occurrences, they largely neglect recovery once an agent encounters such states. Although the latest research has attempted to address this by guiding agents back to in-distribution states, their reliance on uncertainty estimation hinders scalability in complex environments. To overcome this limitation, we introduce Language Models for Out-of-Distribution Recovery (LaMOuR), which enables recovery learning without relying on uncertainty estimation. LaMOuR generates dense reward codes that guide the agent back to a state where it can successfully perform its original task, leveraging the capabilities of LVLMs in image description, logical reasoning, and code generation. Experimental results show that LaMOuR substantially enhances recovery efficiency across diverse locomotion tasks and even generalizes effectively to complex environments, including humanoid locomotion and mobile manipulation, where existing methods struggle. The code and supplementary materials are available at https://lamour-rl.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。