解决VAE在机器人学习中因单样本近似导致的策略不稳定问题
P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning
- 用概率传播框架同时优化均值和方差，提升策略稳定性
- 实验显示数据效率从64.6%提升至96%以上，收敛步数减少超20%
- 适合需要高鲁棒性的基于VAE的强化学习任务
变分自编码器（VAE）广泛用于机器人领域对高维噪声观测进行编码。然而，其随机潜在空间与近端策略优化（PPO）存在本质矛盾：有效策略需对潜在分布进行边际化，而现有方法仅用单一潜在样本估算概率比和KL散度。我们发现根本原因在于潜在空间中朴素的单样本近似会引入显著的方差与偏差。为此，提出P^3（概率策略传播），一种面向基于VAE策略的分布感知优化框架。P^3结合基于矩的概率方法实现稳定高效学习，并通过采样校准增强潜在不确定性下的策略鲁棒性。实验表明，P^3将数据效率从64.6%提升至超过96%，收敛步数减少超过20%。此外，其在复杂的人形机器人越障任务中表现优异，为基于VAE的PPO提供了可靠基础。代码已开源。
原文摘要 · Abstract (English)
Variational Autoencoders are widely used to encode high-dimensional and noisy observations in robotics. However, their stochastic latent creates a mismatch with Proximal Policy Optimization (PPO): an effective policy marginalizes over the latent distribution, whereas former implementations estimate its probability ratio and KL divergence using only one latent sample. We identify a fundamental but overlooked theoretical cause: naive single-sample approximations in stochastic latent space induce significant variance and bias in the surrogate loss. To address this, we introduce P^3 (Probabilistic Policy Propagation), a distribution-aware optimization framework for VAE-based policies. $P^3$ couples moment-based probabilistic method for stable and efficient learning with sampling-based calibration for robust policy behavior under latent uncertainty. In our experiments, P^3 boosts data efficiency from 64.6% to >96%, reduces convergence steps by >20%. Furthermore, P^3 is evaluated on challenging humanoid parkour tasks and shows an effective foundation for VAE-based PPO. Code is available at https://github.com/ylyem9x/P3_Open.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。