提出新方法让智能体在对抗观测下更鲁棒，无需额外环境交互。
Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation
- 将对抗学习重构为软约束优化，避免正反智能体相互依赖。
- 理论证明双方策略评估具有对称性，支持方法稳定性。
- 适合追求高效、安全的强化学习应用，如自动驾驶决策。
近年来，针对对抗性观测输入的鲁棒强化学习方法受到广泛关注，源于强化学习固有的脆弱性。现有方法虽取得一定成效，但在长时程最坏情况下的应对仍需同时最小化对手的累计奖励并训练智能体进行反击，这一过程导致智能体与对手间存在相互依赖，使环境交互效率低下，并阻碍了离策略方法的发展。本文提出一种新型离策略方法，通过将对抗学习重构为软约束优化问题，消除对额外环境交互的需求。该方法基于智能体与对手之间策略评估的对称性，获得理论支持。代码已开源：https://github.com/nakanakakosuke/VALT_SAC。
原文摘要 · Abstract (English)
Recently, robust reinforcement learning (RL) methods designed to handle adversarial input observations have received significant attention, motivated by RL's inherent vulnerabilities. While existing approaches have demonstrated reasonable success, addressing worst-case scenarios over long time horizons requires both minimizing the agent's cumulative rewards for adversaries and training agents to counteract them through alternating learning. However, this process introduces mutual dependencies between the agent and the adversary, making interactions with the environment inefficient and hindering the development of off-policy methods. In this work, we propose a novel off-policy method that eliminates the need for additional environmental interactions by reformulating adversarial learning as a soft-constrained optimization problem. Our approach is theoretically supported by the symmetric property of policy evaluation between the agent and the adversary. The implementation is available at https://github.com/nakanakakosuke/VALT_SAC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。