让机器人在看不见全身状态时也能稳定行走,性能接近全知版本。
Success in Humanoid Reinforcement Learning under Partial Observation
- 用固定长度历史序列并行编码过去观测,补全缺失信息
- 仅用1/3到2/3状态,性能媲美全状态学习的顶尖模型
- 能适应身体质量变化,适合做鲁棒机器人控制研究
强化学习广泛应用于机器人控制,但在高维任务如人形机器人行走中,不完全观测下的有效策略学习仍是重大挑战。此前尚无研究在基准Gymnasium Humanoid-v4环境中实现基于不完整状态信息的稳定训练。该环境目标是尽可能快速前行而不摔倒,奖励来自保持直立和前进,惩罚来自过度动作及外部接触力。本文首次在该环境下成功实现了部分可观测性下的策略学习。所学策略在仅使用原始状态的1/3至2/3的情况下,性能达到与全状态访问的最先进结果相当的水平。此外,策略对机器人属性(如部件质量变化)具有适应能力。成功关键在于一种新型历史编码器,可并行处理固定长度的历史观测序列。将其集成到标准无模型算法中后,性能与全观测基线持平。我们推测其通过重构近期观测中的关键上下文信息,实现稳健决策。
原文摘要 · Abstract (English)
Reinforcement learning has been widely applied to robotic control, but effective policy learning under partial observability remains a major challenge, especially in high-dimensional tasks like humanoid locomotion. To date, no prior work has demonstrated stable training of humanoid policies with incomplete state information in the benchmark Gymnasium Humanoid-v4 environment. The objective in this environment is to walk forward as fast as possible without falling, with rewards provided for staying upright and moving forward, and penalties incurred for excessive actions and external contact forces. This research presents the first successful instance of learning under partial observability in this environment. The learned policy achieves performance comparable to state-of-the-art results with full state access, despite using only one-third to two-thirds of the original states. Moreover, the policy exhibits adaptability to robot properties, such as variations in body part masses. The key to this success is a novel history encoder that processes a fixed-length sequence of past observations in parallel. Integrated into a standard model-free algorithm, the encoder enables performance on par with fully observed baselines. We hypothesize that it reconstructs essential contextual information from recent observations, thereby enabling robust decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。