通过引导高价值动作提升离线强化学习性能
Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
- 用多步流匹配策略结合单步精简演员，定向模仿高价值动作
- 在144个任务上达顶尖表现，尤其在数据差和难任务上提升显著
- 适合研究离线强化学习中策略偏差与数据利用效率的学者
离线强化学习通常依赖行为正则化，使策略保持在数据集分布附近。然而，这类方法无法区分高价值与低价值动作。我们提出引导流策略（GFP），将多步流匹配策略与单步精简演员相结合。演员通过加权行为克隆引导流策略，聚焦于数据集中高价值动作的模仿，而非无差别复制所有状态-动作对。反过来，流策略通过约束演员，使其保持与数据集中最优转移对的一致性，同时最大化评判器得分。这种双向引导使GFP在来自OGBench、Minari和D4RL基准的144个状态和像素级任务上达到当前最优性能，尤其在次优数据集和挑战性任务上表现出显著提升。
原文摘要 · Abstract (English)
Offline reinforcement learning often relies on behavior regularization that enforces policies to remain close to the dataset distribution. However, such approaches fail to distinguish between high-value and low-value actions in their regularization components. We introduce Guided Flow Policy (GFP), which couples a multi-step flow-matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on suboptimal datasets and challenging tasks. Webpage: https://simple-robotics.github.io/publications/guided-flow-policy/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。