用流模型提升离线强化学习的策略表达能力，效果显著。
Flow Q-Learning
- 用单步流模型替代迭代生成，避免训练不稳和推理开销。
- 在73个挑战性任务上表现优异，涵盖状态与像素输入。
- 适合追求高效稳定离线强化学习的开发者或研究者。
我们提出流Q-learning（FQL），一种简单且高效的离线强化学习方法，利用表达能力强的流匹配策略建模数据中任意复杂动作分布。由于动作生成过程具有迭代特性，直接用强化学习训练流策略面临挑战。为此，FQL改用强化学习训练一个表达性强的单步策略，而非直接引导迭代式流策略以最大化价值。该方法完全避免了不稳定的递归反向传播，消除了测试时昂贵的迭代动作生成，同时仍保持高表达能力。实验表明，FQL在73个具有挑战性的状态与像素级任务（OGBench和D4RL）中均取得优异表现，涵盖离线强化学习及离线到在线迁移学习场景。
原文摘要 · Abstract (English)
We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method that leverages an expressive flow-matching policy to model arbitrarily complex action distributions in data. Training a flow policy with RL is a tricky problem, due to the iterative nature of the action generation process. We address this challenge by training an expressive one-step policy with RL, rather than directly guiding an iterative flow policy to maximize values. This way, we can completely avoid unstable recursive backpropagation, eliminate costly iterative action generation at test time, yet still mostly maintain expressivity. We experimentally show that FQL leads to strong performance across 73 challenging state- and pixel-based OGBench and D4RL tasks in offline RL and offline-to-online RL. Project page: https://seohong.me/projects/fql/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。