UNIQ让离线强化学习在数据丰富区更激进，稀疏区更保守。
UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning
- 用分位数校准不确定性，实现状态自适应的保守策略
- 在D4RL MuJoCo上比IQL提升显著，最高增益达Walker2d任务
- 仅需约250MB显存，比EDAC快10倍，适合资源受限场景
离线强化学习需谨慎控制保守性以缓解分布偏移问题，但现有方法对所有状态施加固定惩罚，忽视局部数据覆盖差异。我们提出UNIQ（不确定性感知分位数），一种基于隐式Q学习（IQL）框架的离线强化学习方法，通过分位数校准不确定性估计实现状态自适应保守性。UNIQ训练多期望值集成模型，利用分裂共形预测计算无需分布假设的不确定性估计，并将该信号映射为状态相关的期望值：在数据覆盖良好的区域放松保守性，在数据边界附近增强保守性。在D4RL MuJoCo基准测试中，UNIQ持续优于IQL，尤其在Walker2d及高重放任务上表现突出。同时，其峰值显存消耗约250MB，相比EDAC降低约10倍。本工作不追求整体性能最优，而是聚焦于提升离线强化学习中的性能-效率平衡，提供可实用的机制改进。
原文摘要 · Abstract (English)
Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ (Uncertainty-Informed Quantile), an offline RL method that introduces state-adaptive conservatism through conformally calibrated uncertainty estimation. Built on the Implicit Q-Learning (IQL) backbone, UNIQ trains a multi-expectile value ensemble, computes distribution-free uncertainty estimates using split conformal prediction, and maps the resulting signal to a state-dependent expectile that relaxes conservatism in well-covered regions while strengthening it in uncertain regions near the data frontier. On D4RL MuJoCo benchmarks, UNIQ consistently improves over IQL, with the largest gains observed on Walker2d and replay-heavy tasks. At the same time, UNIQ operates at near-IQL memory cost (approximately 250 MB peak VRAM), providing roughly a 10x reduction compared to EDAC. Rather than pursuing overall state-of-the-art performance, we position UNIQ as a practical mechanism contribution that improves the performance-efficiency trade-off in offline reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。