用平滑优化提升离线强化学习抗数据污染能力
Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
- 引入SAM优化器寻找损失曲面更平坦的解
- 在D4RL上对抗性污染下性能显著超越基线
- 适合关注离线强化学习鲁棒性的研究者
离线强化学习对真实世界的数据污染敏感，即使鲁棒算法在观测污染和混合污染下也会失效。我们认为这源于数据污染导致损失曲面出现尖锐极小值，进而影响泛化能力。为此，我们首次将尖锐感知最小化（Sharpness-Aware Minimization, SAM）作为通用、即插即用的优化器应用于离线强化学习。SAM旨在寻找更平坦的极小值，引导模型进入更鲁棒的参数区域。我们将SAM集成到两种强基线中：IQL——当前该场景下表现最佳的离线强化学习算法；以及专门设计用于数据污染鲁棒性的RIQL。在加入随机与对抗性污染的D4RL基准测试中，增强后的算法持续且显著优于原始基线。奖励曲面可视化显示SAM找到了更平滑的解，有力证明其在提升离线强化学习鲁棒性方面的有效性。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) is vulnerable to real-world data corruption, with even robust algorithms failing under challenging observation and mixture corruptions. We posit this failure stems from data corruption creating sharp minima in the loss landscape, leading to poor generalization. To address this, we are the first to apply Sharpness-Aware Minimization (SAM) as a general-purpose, plug-and-play optimizer for offline RL. SAM seeks flatter minima, guiding models to more robust parameter regions. We integrate SAM into strong baselines for data corruption: IQL, a top-performing offline RL algorithm in this setting, and RIQL, an algorithm designed specifically for data-corruption robustness. We evaluate them on D4RL benchmarks with both random and adversarial corruption. Our SAM-enhanced methods consistently and significantly outperform the original baselines. Visualizations of the reward surface confirm that SAM finds smoother solutions, providing strong evidence for its effectiveness in improving the robustness of offline RL agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。