用谱归一化让机器人走路策略更稳定,同时节省显存。
Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion
- 用权重谱范数约束替代梯度惩罚,实现平滑控制
- 在仿真和真实机器人上性能接近传统方法,显存减少超60%
- 适合需高效训练高维动作策略的机器人研究者
强化学习在训练类人机器人复杂步态方面展现出巨大潜力,但仿真中训练的策略常因不现实假设(如无限执行器带宽、无扭矩限制)而无法迁移到真实机器人。这些假设使策略依赖高频、剧烈的扭矩变化,超出实际执行器能力。传统方法通过正则化项(如关节速度、加速度、能耗)惩罚激进运动,但需大量调参;而唇部连续性策略(LCP)虽能限制带宽,却因依赖梯度计算导致显著显存开销。本文提出谱归一化(SN),通过约束网络权重的谱范数,有效抑制高频策略波动,同时大幅降低显存占用。在仿真与真实类人机器人上的实验表明,SN性能接近梯度惩罚方法,且支持更高效的并行训练。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has shown great potential in training agile and adaptable controllers for legged robots, enabling them to learn complex locomotion behaviors directly from experience. However, policies trained in simulation often fail to transfer to real-world robots due to unrealistic assumptions such as infinite actuator bandwidth and the absence of torque limits. These conditions allow policies to rely on abrupt, high-frequency torque changes, which are infeasible for real actuators with finite bandwidth. Traditional methods address this issue by penalizing aggressive motions through regularization rewards, such as joint velocities, accelerations, and energy consumption, but they require extensive hyperparameter tuning. Alternatively, Lipschitz-Constrained Policies (LCP) enforce finite bandwidth action control by penalizing policy gradients, but their reliance on gradient calculations introduces significant GPU memory overhead. To overcome this limitation, this work proposes Spectral Normalization (SN) as an efficient replacement for enforcing Lipschitz continuity. By constraining the spectral norm of network weights, SN effectively limits high-frequency policy fluctuations while significantly reducing GPU memory usage. Experimental evaluations in both simulation and real-world humanoid robot show that SN achieves performance comparable to gradient penalty methods while enabling more efficient parallel training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。