用流模型加隐变量引导，让强化学习策略更强大且稳定。
FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
- 引入流模型与隐变量扩展状态空间，避免重要性权重崩溃。
- 仅用少量采样样本即实现高维动作空间的高效策略优化。
- 在多个复杂任务上达到当前最优性能，适合高维控制场景。
最大熵强化学习（MaxEnt-RL）能实现鲁棒探索，但实际应用中常将策略限制为简单的高斯分布。近期方法虽通过重要性加权监督学习引入表达性强的生成策略，却易出现重要性权重坍缩，难以在高维动作空间中扩展。本文核心洞察是通过局部化采样区域，避免在整个动作空间上使用重要性采样引发的权重退化。为此提出 extbf{FLAG}（Flow policy with Latent-Augmented Guidance）：在状态空间中加入流隐变量，优化一个可证明一致的代理最大熵强化学习目标。实验表明，FLAG 能在有限重要性样本下实现表达性策略优化，并成功扩展至高维控制任务。此外，FLAG 在多个挑战性基准上取得当前最优性能。
原文摘要 · Abstract (English)
Maximum entropy reinforcement learning (MaxEnt-RL) enables robust exploration, yet practical implementations often restrict policies to simple Gaussians. While recent approaches incorporate expressive generative policies via importance-weighted supervised learning, they are prone to importance weight collapse, which limits their scalability in high-dimensional action spaces. Our key insight is to mitigate this limitation by localizing the sampling region, avoiding the weight degeneracy induced by importance sampling over the entire action space. To instantiate this insight, we introduce \textbf{FLAG} (\textbf{F}low policy with \textbf{L}atent-\textbf{A}ugmented \textbf{G}uidance). FLAG augments the state space with a flow latent variable and optimizes a provably consistent proxy MaxEnt-RL objective. We empirically demonstrate that FLAG enables expressive policy optimization with limited importance samples and scales to high-dimensional control tasks. Furthermore, FLAG achieves state-of-the-art performance across challenging benchmarks. Our project webpage: https://flag-rl.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。