用流模型提升强化学习表达力，同时保持训练稳定。
Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy
- 利用流模型的确定性特性，将终态价值回传至中间隐状态。
- 在OGBench上平均性能比顶尖基线高出10.6个百分点。
- 适合需要高表达力且稳定训练的强化学习任务。
由于流模型具有强大的表达能力，其作为强化学习决策策略的研究日益增多。然而，有效利用这种表达能力以实现价值最大化仍具挑战性，因为直接基于梯度的优化需对数值求解器进行反向传播，常导致训练不稳定。现有方法通常通过限制流模型的表达能力来解决此问题，造成优化稳定性与表征灵活性之间的权衡。为此，我们提出Q-Flow框架，利用流动力学的确定性，沿策略诱导的流显式地将终态轨迹价值传播至中间隐状态。该形式化使无需展开数值求解器即可使用中间值梯度进行稳定策略优化，有效弥合了稳定性与表达力之间的差距。我们在具有挑战性的OGBench基准上评估Q-Flow，其在离线学习设置中平均性能优于现有最优基线10.6个百分点，同时在同一框架内实现了稳定的在线适应。
原文摘要 · Abstract (English)
There is growing interest in utilizing flow-based models as decision-making policies in reinforcement learning due to their high expressive capacity. However, effectively leveraging this expressivity for value maximization remains challenging, as naive gradient-based optimization requires backpropagating through numerical solvers and often leads to instability. Existing approaches typically address this issue by restricting the expressive capacity of flow-based policies, resulting in a trade-off between optimization stability and representational flexibility. To resolve this, we introduce Q-Flow, a framework that leverages the deterministic nature of flow dynamics to explicitly propagate terminal trajectory value to intermediate latent states along the policy-induced flow. This formulation enables stable policy optimization using intermediate value gradients without unrolling the numerical solver, effectively bridging the gap between stability and expressivity. We evaluate Q-Flow in the offline learning setting on the challenging OGBench suite, where it consistently outperforms state-of-the-art baselines by an average of 10.6 percentage points, while also enabling stable online adaptation within the same framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。