提出自适应约束缩放方法，让离线强化学习无需调参就能稳定高效训练。
Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
- 通过二阶可微框架动态平衡强化学习与行为克隆
- 39个数据集上仅用单一超参数即超越需调参的先进算法
- 适合追求少调参、高鲁棒性的离线强化学习研究者
离线强化学习（Offline RL）可在不与环境交互的情况下，仅从固定数据集中学习有效策略。现有方法通常使用策略约束来缓解训练过程中的分布偏移问题。然而，由于约束尺度随任务和数据质量变化，现有方法必须精细调整超参数以适配每组数据，耗时且常不切实际。本文提出自适应约束缩放（ASPC），一种二阶可微框架，可在训练中动态平衡强化学习与行为克隆（BC）。我们理论分析了其性能提升保证。在四个D4RL领域共39个数据集上的实验表明，ASPC采用单一超参数配置即可超越其他自适应约束方法，并优于需要逐数据集调参的最先进离线强化学习算法，同时计算开销极低。代码将发布于https://github.com/Colin-Jing/ASPC。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) enables learning effective policies from fixed datasets without any environment interaction. Existing methods typically employ policy constraints to mitigate the distribution shift encountered during offline RL training. However, because the scale of the constraints varies across tasks and datasets of differing quality, existing methods must meticulously tune hyperparameters to match each dataset, which is time-consuming and often impractical. We propose Adaptive Scaling of Policy Constraints (ASPC), a second-order differentiable framework that dynamically balances RL and behavior cloning (BC) during training. We theoretically analyze its performance improvement guarantee. In experiments on 39 datasets across four D4RL domains, ASPC using a single hyperparameter configuration outperforms other adaptive constraint methods and state-of-the-art offline RL algorithms that require per-dataset tuning while incurring only minimal computational overhead. The code will be released at https://github.com/Colin-Jing/ASPC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。