解决离线强化学习中状态分布外问题，提升模型鲁棒性。
Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
- 统一修正分布外状态并抑制分布外动作
- 在标准基准上表现优异，无需调参
- 增强对环境扰动的鲁棒性，适合实际部署
在离线强化学习中，分布外（OOD）动作问题备受关注，但我们认为分布外状态问题同样影响性能却未被充分研究。该问题表现为测试阶段代理遇到离线数据集之外的状态，导致行为失控和性能下降。为此，我们提出SCAS方法，将分布外状态修正与分布外动作抑制统一整合。技术上，SCAS实现基于价值感知的分布外状态修正，能将代理从分布外状态引导至高价值的分布内状态。理论与实证结果表明，SCAS同时具备抑制分布外动作的效果。在标准离线强化学习基准上，SCAS表现出色且无需额外超参数调优。此外，得益于其分布外状态修正能力，SCAS在面对环境扰动时展现出更强的鲁棒性。
原文摘要 · Abstract (English)
In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。