基于逆向生成流的离线强化学习算法，提升策略训练效率与性能。
Reversal Q-Learning
- 将生成流步骤视为动作，构建扩展MDP框架
- 通过逆向生成虚拟轨迹，实现对先验数据的离线训练
- 避免时间反向传播，适合复杂机器人任务
迭代生成建模技术（如流匹配）为有效离线强化学习提供了强大工具。本文提出一种新的无偏策略强化学习算法——逆向Q学习（Reversal Q-learning, RQL），基于先验数据训练流策略。核心思想源自扩展马尔可夫决策过程（MDP）框架，将每次流精炼步骤视为独立动作。为在该框架内实现离线策略学习，我们采用两种技术：通过逆向流生成虚拟在线轨迹以兼容先验数据，并应用偏差-方差降低技术缓解离线学习中的时序灾难问题。实验在50个具有挑战性的模拟机器人任务上验证，RQL在离线强化学习平均表现上优于现有最先进的基于流的方法，且不依赖时间反向传播，更高效利用价值函数，直接训练完整表达力强的流策略。
原文摘要 · Abstract (English)
Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expanded" Markov decision process (MDP) framework, which treats individual flow refinement steps as separate actions in an MDP. To enable off-policy RL within this framework, we apply two techniques: we generate virtual on-policy trajectories (by "reversing" flows) to make this framework compatible with prior data, and we apply a bias-and-variance reduction technique to mitigate the curse of horizon in off-policy RL. We call the resulting algorithm Reversal Q-learning (RQL). RQL has several advantages over previous flow-based RL methods: it does not suffer from backpropagation through time, makes better use of the learned value function, and directly trains the full, expressive flow policy. Through our experiments on 50 challenging simulated robotic tasks, we show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。