提出悲观辅助策略，减少离线强化学习中的动作误差累积。
Pessimistic Auxiliary Policy for Offline Reinforcement Learning
- 构建悲观辅助策略，通过最大化Q函数下置信界来选择可靠动作。
- 在策略附近保持高价值与低不确定性，避免采样高估错误动作。
- 可显著提升多种离线RL方法性能，适合追求稳定性的研究者。
离线强化学习旨在从预收集的数据集中学习智能体，避免实时交互带来的不安全和低效问题。然而，学习过程中不可避免地会接触到分布外的动作，引发近似误差，导致误差累积和显著高估。本文构建了一种新的悲观辅助策略，用于采样可靠的行动。具体而言，通过最大化Q函数的下置信界，设计悲观辅助策略。该策略在已学习策略附近表现出较高价值与较低不确定性，避免了学习过程中采样高价值但可能误差较大的动作。由悲观辅助策略采样的动作引入的近似误差更小，从而缓解了误差累积问题。在多个离线强化学习基准测试中，实验表明使用该悲观辅助策略能有效提升其他离线强化学习方法的性能。
原文摘要 · Abstract (English)
Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。