用优势函数筛选离域动作，提升离线强化学习泛化能力
Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
- 通过状态价值间接评估动作优劣，动态调节Q值更新
- 在D4RL基准上实现当前最优性能，尤其在困难任务中提升显著
- 适合需要安全探索和高可靠性策略的离线强化学习场景
离线强化学习从固定数据集学习策略，但面临分布偏移问题，导致对离域（OOD）动作的评估不准确并产生过估计。现有方法保守抑制所有离域动作，限制了泛化能力。本文提出基于优势的扩散演员-评论家（ADAC），通过类似优势的函数评估离域动作，并据此差异化地调节Q函数更新。核心洞察是状态值函数通常比动作值函数更可靠，因此利用下一状态值间接评估每个动作。我们在PointMaze环境中可视化显示，该优势调制能有效选择优质离域动作，同时排除劣质动作。在D4RL基准上的大量实验表明，ADAC达到当前最优性能，尤其在挑战性任务中表现突出。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) learns policies from fixed datasets without online interactions, but suffers from distribution shift, causing inaccurate evaluation and overestimation of out-of-distribution (OOD) actions. Existing methods counter this by conservatively discouraging all OOD actions, which limits generalization. We propose Advantage-based Diffusion Actor-Critic (ADAC), which evaluates OOD actions via an advantage-like function and uses it to modulate the Q-function update discriminatively. Our key insight is that the (state) value function is generally learned more reliably than the action-value function; we thus use the next-state value to indirectly assess each action. We develop a PointMaze environment to clearly visualize that advantage modulation effectively selects superior OOD actions while discouraging inferior ones. Moreover, extensive experiments on the D4RL benchmark show that ADAC achieves state-of-the-art performance, with especially strong gains on challenging tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。