通过分块采样与梯度修正，缓解离线强化学习中的分布偏移问题。
Less is More: Clustered Cross-Covariance Control for Offline RL
- 将经验回放缓冲区分块，限制更新范围以减少异常协方差影响。
- 在小数据和高分布外场景下，回报提升最高达30%。
- 方法易集成，适合数据稀缺或分布偏移严重的离线强化学习任务。
离线强化学习的核心挑战是分布偏移问题。数据稀疏或数据集被分布外（OOD）区域主导时，该问题加剧。我们的理论分析与实验表明，标准平方误差目标会引入有害的时序差分（TD）交叉协方差，该效应在分布外区域被放大，导致优化偏差并损害策略学习。为对抗此机制，我们提出两种互补策略：分块缓冲区采样，将更新限制在局部回放缓冲区，抑制不规则协方差影响并对齐更新方向，形成易于集成的方案——用于TD的聚类交叉协方差控制（C^4）。同时引入基于梯度的显式校正惩罚，消除每次更新中的协方差偏差。我们证明，缓冲区分块能保持最大化目标的下界性质，且这些约束在极端分布外区域不会引入过度保守性，同时保留策略约束离线强化学习的核心行为。实验表明，本方法在稳定性与回报上均优于先前方法，尤其在小数据集及强调分布外区域的数据划分下，回报最高提升30%。
原文摘要 · Abstract (English)
A fundamental challenge in offline reinforcement learning is distributional shift. Scarce data or datasets dominated by out-of-distribution (OOD) areas exacerbate this issue. Our theoretical analysis and experiments show that the standard squared error objective induces a harmful TD cross covariance. This effect amplifies in OOD areas, biasing optimization and degrading policy learning. To counteract this mechanism, we develop two complementary strategies: partitioned buffer sampling that restricts updates to localized replay partitions, attenuates irregular covariance effects, and aligns update directions, yielding a scheme that is easy to integrate with existing implementations, namely Clustered Cross-Covariance Control for TD (C^4). We also introduce an explicit gradient-based corrective penalty that cancels the covariance induced bias within each update. We prove that buffer partitioning preserves the lower bound property of the maximization objective, and that these constraints mitigate excessive conservatism in extreme OOD areas without altering the core behavior of policy constrained offline reinforcement learning. Empirically, our method showcases higher stability and up to 30% improvement in returns over prior methods, especially with small datasets and splits that emphasize OOD areas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。