用确定性潜空间引导生成策略，实现稳定精准的机器人控制
Steering Generative Reinforcement Learning into Stable Robotic Controller
- 训练时用生成模型随机探索，部署时由学习的潜空间控制器确定性输出动作
- 在6个Isaac Lab任务和G1机器狗行走任务中均优于经典强化学习与生成式强化学习基线
- 适合需要高稳定性与精确响应的复杂机器人控制场景
扩散和基于流的生成策略通过迭代动作生成提供丰富的随机探索，是强化学习中的强大策略类。然而，扩散策略的随机性不适用于高维机器人系统的稳定精确控制，因微小动作变化会累积导致运动不一致和鲁棒性下降。为此，我们提出SteerGenPO，一种潜空间强化学习框架，将训练好的生成策略引导为稳健的确定性机器人控制器。核心思想是用学习的潜空间演员替代生成策略的随机潜变量采样，该演员根据状态预测一个状态依赖的潜输入。此方法分离了探索与控制：训练阶段通过随机生成采样实现多样化动作提议，部署阶段则通过确定性潜空间引导实现稳定自适应控制。我们在六个Isaac Lab基准任务和Unitree G1步态任务上评估了SteerGenPO。结果表明，其性能优于经典强化学习与生成式强化学习基线，且确定性潜空间引导在推理时产生更稳定的行为和更可靠的指令响应。
原文摘要 · Abstract (English)
Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich stochastic exploration through iterative action generation. However, the stochasticity of diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control: stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。