用归一化流提升机器人策略学习的多模态建模能力
NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning
- 用归一化流替代传统高斯分布建模策略，支持多模态分布
- 在多个仿真环境中实现稳定训练，性能超越传统方法
- 适合需要复杂动作策略的机器人任务，尤其关注实际部署
深度强化学习近年发展迅速，广泛应用于各类场景。但在基于DRL的机器人策略学习中，当前主流仍采用对角协方差的多元高斯分布参数化策略，难以建模多模态分布。本文探索将现代网络结构——归一化流（Normalizing Flow, NF）作为策略参数化方式，因其具备多模态建模能力、可解析计算对数概率以及低计算与内存开销。然而，直接在在线强化学习中训练NF常导致训练不稳定。本文深入分析该现象，并提出简单有效的解决方案。通过在多个仿真环境中的大量实验，验证所提方法NFPO在常见机器人学习任务中表现稳健且强大，成功实现向真实机器人迁移。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。