提出新方法控制离线强化学习中异常动作的修正传播,提升价值估计稳定性。
Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL

- 用凸包邻域平滑双泛化框架分离样本内参考与异常动作修正
- 在MuJoCo和AntMaze上实现稳定价值估计和优异综合性能
- 无需悲观惩罚或精确分类,适合追求鲁棒性的离线强化学习研究者
离线强化学习可从邻近的分布外(OOD)动作中受益,但这些动作的估计误差可能因自举机制被放大。现有正则化与局部泛化方法通常分别控制允许的OOD区域或泛化目标的影响,使用独立机制。本文提出凸包邻域平滑双泛化(CSDG),将贝尔曼更新表达为样本内价值目标加上一个基于凸包邻域的局部修正项。该形式使泛化贡献显式可见,并将其与样本内参考路径分离。修正项通过在不同扰动半径下采样的样本内与分布外候选值进行平滑获得,混合系数λ控制其在每步更新中的贡献,递归折扣率保持为γ。在有界性与固定扰动核条件下,我们推导出精确的一步修正恒等式、时变迭代上界,以及仅依赖于固定点处分支差异的不动点上界。进一步刻画了理想算子诱导的隐式策略,并给出了条件非退化准则。实用算法采用非对称有界噪声和期望分位回归近似这些量,无需精确支持分类或额外的悲观性OOD惩罚。在Gym-MuJoCo和AntMaze上的实验显示其具有强综合表现和稳定的值估计能力。代码见:https://github.com/YOUNG-fnxm/CSDG
原文摘要 · Abstract (English)
Offline reinforcement learning (offline RL) can benefit from nearby out-of-distribution (OOD) actions, but estimation errors at these actions may be amplified by bootstrapping. Existing regularization and local-generalization methods control either the admissible OOD region or the influence of generalized targets, often through separate mechanisms. We propose Convex Hull Neighborhood Smooth Dual Generalization (CSDG), which expresses the Bellman backup as an in-sample value target plus a CHN-local correction. This formulation makes the generalized contribution explicit and separates it from the in-sample reference path. The correction is obtained by smoothing in-sample-oriented and OOD-oriented candidates sampled at different perturbation radii. A mixture coefficient lambda scales its contribution to each backup, while the recursive discount remains gamma. Under boundedness and fixed perturbation kernels, we derive an exact one-step correction identity, a time-varying iterate bound, and a fixed-point bound that depends only on the branch discrepancy at the fixed point. We further characterize the implicit policies induced by the idealized operators and give a conditional non-degradation criterion. The practical algorithm approximates these quantities using asymmetric bounded noise and expectile regression, without exact support classification or an additional pessimistic OOD penalty. Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation. Code is available at: https://github.com/YOUNG-fnxm/CSDG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。