arXiv:2501.17667cs.LGcs.CR2025-01中稿 · USENIX Security Sy…被引 2

提升强化学习的抗干扰能力,让智能体在攻击下仍能稳定发挥。

CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximization

  • 通过优化局部认证半径来增强全局抗干扰能力
  • 相比基线方法,认证回报最高提升一倍
  • 适合关注安全可控强化学习的研究者

深度强化学习(DRL)在动态环境中的控制与决策任务中表现优异,但对观测噪声和对抗攻击敏感,其鲁棒性备受关注。现有方法多依赖简单高斯增强训练策略,导致认证鲁棒性与收益之间权衡不佳。为此,本文提出一种新范式—— exttt{CAMP}(Certified-Radius-Maximizing Policy),利用训练时统计量将全局认证半径归约为局部半径的函数,构建代理损失并优化策略。同时引入策略模仿技术以稳定训练过程。实验表明, exttt{CAMP} 在多个任务上显著改善了鲁棒性-收益权衡,认证期望回报最高可达基线两倍。

原文摘要 · Abstract (English)

Deep reinforcement learning (DRL) has gained widespread adoption in control and decision-making tasks due to its strong performance in dynamic environments. However, DRL agents are vulnerable to noisy observations and adversarial attacks, and concerns about the adversarial robustness of DRL systems have emerged. Recent efforts have focused on addressing these robustness issues by establishing rigorous theoretical guarantees for the returns achieved by DRL agents in adversarial settings. Among these approaches, policy smoothing has proven to be an effective and scalable method for certifying the robustness of DRL agents. Nevertheless, existing certifiably robust DRL relies on policies trained with simple Gaussian augmentations, resulting in a suboptimal trade-off between certified robustness and certified return. To address this issue, we introduce a novel paradigm dubbed \texttt{C}ertified-r\texttt{A}dius-\texttt{M}aximizing \texttt{P}olicy (\texttt{CAMP}) training. \texttt{CAMP} is designed to enhance DRL policies, achieving better utility without compromising provable robustness. By leveraging the insight that the global certified radius can be derived from local certified radii based on training-time statistics, \texttt{CAMP} formulates a surrogate loss related to the local certified radius and optimizes the policy guided by this surrogate loss. We also introduce \textit{policy imitation} as a novel technique to stabilize \texttt{CAMP} training. Experimental results demonstrate that \texttt{CAMP} significantly improves the robustness-return trade-off across various tasks. Based on the results, \texttt{CAMP} can achieve up to twice the certified expected return compared to that of baselines. Our code is available at https://github.com/NeuralSec/camp-robust-rl.

强化学习鲁棒性认证策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。