用分位数回归优化离线强化学习中的温度系数，提升稳定性和泛化性。
Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
- 通过分位数回归自动估计温度系数β，减少对超参数的依赖
- 在D4RL和NeoRL2上表现优于或相当现有方法，训练更稳定
- 适合需要高稳定性与少调参的工业级离线强化学习场景
离线强化学习（Offline RL）允许仅从固定数据集学习策略而无需与环境进一步交互，特别适用于高风险或高成本领域。极端Q学习（XQL）是一种近期的离线RL方法，利用极值定理建模贝尔曼误差，表现出优异的实证性能。然而，XQL及其稳定版本MXQL存在明显局限：需针对每个数据集和领域进行大量超参数调优，且训练过程不稳定。为此，我们提出一种基于温和假设下分位数回归的温度系数β的合理估计方法。为进一步提升训练稳定性，引入一种具有轻微泛化能力的价值正则化技术，受最近约束价值学习进展启发。实验表明，所提算法在一系列基准任务（包括D4RL和NeoRL2）中实现竞争性或更优性能，同时保持稳定的训练动态，并在所有数据集和领域使用一致的超参数设置。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme $Q$-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient $β$ via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。