用归一化流统一建模策略、价值函数与状态分布，提升强化学习性能。
Normalizing Flows are Capable Models for Continuous Control
- 设计单一归一化流架构，同时支持策略、Q值和状态分布建模
- 在离线、目标条件和无监督强化学习中均实现更高性能
- 无需求解微分方程或使用自回归结构，计算更高效
现代强化学习算法借助强大概率模型（如Transformer、基于能量的模型、扩散/流模型）取得成功。但这些模型常需付出代价：扩散模型表达能力强但计算密集，自回归Transformer可扩展但需离散化表示。归一化流（NFs）则提供更优选择——无需求解微分方程或自回归结构即可实现似然估计与采样。然而其在强化学习中的潜力被低估，部分源于认为其表达能力不足。本文证明此观点错误。基于最新NF进展，我们提出一种单一NF架构，可无缝集成至强化学习算法中，统一建模策略、Q函数与占用度量。该方法简化算法设计，在模仿学习、离线、目标条件及无监督强化学习任务中均取得更优表现。
原文摘要 · Abstract (English)
Modern reinforcement learning (RL) algorithms have found success by using powerful probabilistic models, such as transformers, energy-based models, and diffusion/flow-based models. To this end, RL researchers often choose to pay the price of accommodating these models into their algorithms -- diffusion models are expressive, but are computationally intensive due to their reliance on solving differential equations, while autoregressive transformer models are scalable but typically require learning discrete representations. Normalizing flows (NFs), by contrast, seem to provide an appealing alternative, as they enable likelihoods and sampling without solving differential equations or autoregressive architectures. However, their potential in RL has received limited attention, partly due to the prevailing belief that normalizing flows lack sufficient expressivity. We show that this is not the case. Building on recent work in NFs, we propose a single NF architecture which integrates seamlessly into RL algorithms, serving as a policy, Q-function, and occupancy measure. Our approach leads to much simpler algorithms, and achieves higher performance in imitation learning, offline, goal conditioned RL and unsupervised RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。