针对离线到在线强化学习中的数据污染问题，提出新方法提升鲁棒性。
Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
- 用逆概率加权缓解数据污染导致的策略重尾问题
- 在D4RL多个污染场景下达到当前最优性能
- 适合部署于真实环境且数据易受干扰的强化学习任务
先在离线数据上预训练策略，再通过在线交互微调的离线到在线强化学习（O2O RL）是实际应用中极具前景的范式。然而，现实环境中离线数据集和在线交互常存在噪声甚至恶意污染，严重损害O2O RL性能。现有工作多聚焦于通过在线探索缓解离线策略的保守性，但对数据污染（包括状态、动作、奖励和动态）下的鲁棒性仍缺乏研究。本文发现数据污染会导致策略出现重尾行为，显著降低在线探索效率。为此，引入逆概率加权（IPW）以减轻重尾现象，提出简单而有效的新方法RPEX（Robust Policy EXpansion）。在D4RL数据集上的大量实验表明，RPEX在多种数据污染场景下均取得当前最优（SOTA）的O2O性能。代码已公开于https://github.com/felix-thu/RPEX。
原文摘要 · Abstract (English)
Pretraining a policy on offline data followed by fine-tuning through online interactions, known as Offline-to-Online Reinforcement Learning (O2O RL), has emerged as a promising paradigm for real-world RL deployment. However, both offline datasets and online interactions in practical environments are often noisy or even maliciously corrupted, severely degrading the performance of O2O RL. Existing works primarily focus on mitigating the conservatism of offline policies via online exploration, while the robustness of O2O RL under data corruption, including states, actions, rewards, and dynamics, is still unexplored. In this work, we observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To address this issue, we incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness, and propose a novel, simple yet effective method termed $\textbf{RPEX}$: $\textbf{R}$obust $\textbf{P}$olicy $\textbf{EX}$pansion. Extensive experimental results on D4RL datasets demonstrate that RPEX achieves SOTA O2O performance across a wide range of data corruption scenarios. Code is available at $\href{https://github.com/felix-thu/RPEX}{https://github.com/felix-thu/RPEX}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。