用记忆防止机器人重复摔倒，提升强化学习样本效率
Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory
- 通过记忆模块存储失败经验，避免重复犯错
- 在MuJoCo任务上提升33.11%样本效率
- 适合需要长周期探索的机器人控制场景
强化学习在机器人学习中取得显著进展。但在复杂探索与高接触动态环境下，早期训练常因碰撞和跌倒等提前终止而受阻。这导致学习被短程、低回报轨迹主导，阻碍收敛并限制长期探索。为此，我们提出失败情景记忆预警（FEMA）技术。FEMA通过情景记忆模块显式存储短程失败经验，在交互过程中检索相似失败经历，防止机器人反复陷入不稳定状态，引导策略走向具有更高长期价值的长程轨迹。FEMA可轻松集成至无模型强化学习算法，使多个经典RL算法在MuJoCo任务上的样本效率提升33.11%。此外，将其应用于并行化PPO训练流水线，验证了其在真实双足机器人任务中的有效性。
原文摘要 · Abstract (English)
Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。