提出稳定训练流模型策略的新方法，解决强化学习中的梯度问题。
SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
- 用速度重参数化重构流模型，避免多步采样梯度崩溃。
- 在连续控制和机器人任务上达到顶尖性能，无需政策蒸馏等技巧。
- 适合研究稳定强化学习算法的学者与工业界应用开发者。
基于流的策略在离线强化学习中因多步动作采样过程的梯度病态而长期存在不稳定性。我们发现，流展开在代数上等价于残差递归计算，因此易受与RNN相同的梯度消失和爆炸影响。为此，我们借鉴现代序列模型原理，对速度网络进行重参数化，提出两种稳定架构：Flow-G（引入门控速度）和Flow-T（采用解码速度）。进而设计一种基于SAC的实用算法，通过噪声增强的轨迹滚动实现端到端训练。该方法支持从零训练和离线到在线迁移，在连续控制与机器人操作基准上达到当前最优表现，彻底摆脱政策蒸馏或代理目标等常见补救手段。
原文摘要 · Abstract (English)
Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process. We trace this instability to a fundamental connection: the flow rollout is algebraically equivalent to a residual recurrent computation, making it susceptible to the same vanishing and exploding gradients as RNNs. To address this, we reparameterize the velocity network using principles from modern sequential models, introducing two stable architectures: Flow-G, which incorporates a gated velocity, and Flow-T, which utilizes a decoded velocity. We then develop a practical SAC-based algorithm, enabled by a noise-augmented rollout, that facilitates direct end-to-end training of these policies. Our approach supports both from-scratch and offline-to-online learning and achieves state-of-the-art performance on continuous control and robotic manipulation benchmarks, eliminating the need for common workarounds like policy distillation or surrogate objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。