FAN算法用单次采样提升离线强化学习效率与性能。
Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
- 用单次流模型迭代和一次噪声采样简化计算
- 在机器人操控与运动任务中实现顶尖性能
- 适合追求高效高精度离线RL的开发者
我们提出Flow-Anchored Noise-conditioned Q-Learning（FAN），一种高效且高性能的离线强化学习算法。近期研究表明，表达性强的流策略和分布式评论家能提升离线RL性能，但计算开销大：流策略需多次采样生成单个动作，分布式评论家需对多个样本（如分位数）进行计算以估计价值。为解决这些低效问题并保持高性能，FAN引入行为正则化技术，仅需一次流策略迭代和一次高斯噪声采样即可完成分布式评论家计算。理论分析表明，这些简化不仅提升效率，还带来更优的任务表现。在机器人操控与运动任务上的实验显示，FAN实现了当前最优性能，同时显著降低训练与推理时间。代码已开源：https://github.com/brianlsy98/FAN。
原文摘要 · Abstract (English)
We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。