DriftQL用单次前向传播实现高效离线强化学习，避免分布外动作。
Drift Q-Learning
- 基于漂移正则化，单网络统一训练，无需迭代去噪
- 在D4RL和OGBench上优于扩散与流模型，数据退化时仍稳定
- 适合追求效率与鲁棒性的离线强化学习应用
离线强化学习需从固定数据中优化策略，同时避免分布外动作带来的不可靠价值估计。扩散与流模型通过建模行为分布来调节强化学习目标，但需迭代去噪、求解器集成，或推理时使用蒸馏等近似方法。我们提出DriftQL，将基于漂移的行为正则化与批评者驱动的策略改进结合：价值信号引导策略趋向高价值区域，吸引与排斥机制共同保持生成动作贴近数据分布，防止模式崩溃。DriftQL以单一网络实现，拥有统一训练目标，单次前向传播生成动作。在D4RL和OGBench上，其性能持续超越扩散与流方法，达到新基准。当数据质量下降时，基线明显失效，而DriftQL仍保持接近清洁数据下的表现，是扩散与流方法的有力替代，兼具确定性方法的简洁与高效。项目页：https://driftql.github.io/
原文摘要 · Abstract (English)
Offline reinforcement learning requires improving a policy from fixed data while avoiding out-of-distribution actions with unreliable value estimates. Diffusion and flow policies handle this trade-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference. We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement. The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean-data performance, positioning it as a promising alternative to diffusion and flow-based methods while maintaining the simplicity and efficiency of deterministic approaches. Project page: https://driftql.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。