提出PIQL改进离线强化学习，解决动作外推误差问题
PIQL: Projective Implicit Q-Learning with Support Constraint for Offline Reinforcement Learning
- 用投影机制替代固定期望值参数，提升适应性
- 在D4RL和NeoRL2上实现当前最优性能
- 适合需要稳定离线强化学习的场景
离线强化学习面临因分布外动作导致的外推误差。隐式Q学习（IQL）通过期望值回归实现样本内学习，但依赖固定期望值超参数和基于密度的策略优化方法，限制了其适应性和性能。本文提出投影隐式Q学习（PIQL），引入支持约束的投影变体。在策略评估阶段，用基于投影的参数替代固定期望值超参数，并将单步价值估计扩展为多步形式；在策略改进阶段，采用支持约束而非密度约束，与策略评估更一致。理论上证明了PIQL保持期望值回归和样本内学习框架，保证单调策略改进，并对优势动作提出逐步严格的判断标准。在D4RL和NeoRL2基准测试中，跨多个领域均实现显著提升，整体达到当前最优性能。
原文摘要 · Abstract (English)
Offline Reinforcement Learning (RL) faces a fundamental challenge of extrapolation errors caused by out-of-distribution (OOD) actions. Implicit Q-Learning (IQL) employs expectile regression to achieve in-sample learning. Nevertheless, IQL relies on a fixed expectile hyperparameter and a density-based policy improvement method, both of which impede its adaptability and performance. In this paper, we propose Projective IQL (PIQL), a projective variant of IQL enhanced with a support constraint. In the policy evaluation stage, PIQL substitutes the fixed expectile hyperparameter with a projection-based parameter and extends the one-step value estimation to a multi-step formulation. In the policy improvement stage, PIQL adopts a support constraint instead of a density constraint, ensuring closer alignment with the policy evaluation. Theoretically, we demonstrate that PIQL maintains the expectile regression and in-sample learning framework, guarantees monotonic policy improvement, and introduces a progressively more rigorous criterion for advantageous actions. Experiments on D4RL and NeoRL2 benchmarks demonstrate robust gains across diverse domains, achieving state-of-the-art performance overall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。