arXiv:2601.10407cs.LG2026-01

针对离线强化学习设计隐蔽性强的后门攻击,仅用5%数据污染即能成功破坏安全约束模型。

CS-GBA: A Critical Sample-based Gradient-guided Backdoor Attack for Offline Reinforcement Learning

  • 基于时序差分误差筛选关键样本,精准分配攻击预算。
  • 在D4RL上以5%污染率实现对CQL等模型的高成功率攻击。
  • 适合研究模型安全性的研究人员,尤其关注对抗性攻击场景。

离线强化学习虽可从静态数据集优化策略,但易受后门攻击。现有方法因随机污染效率低且触发器易被检测而受限。本文提出CS-GBA(关键样本梯度引导后门攻击),通过理论分析发现高时序差分(TD)误差样本对价值函数收敛至关重要,提出自适应关键样本选择机制,将攻击预算集中于最具影响力的状态转移。为规避分布外(OOD)检测,设计相关性破坏触发机制,利用状态特征的物理互斥性(如95%分位边界)保持统计隐蔽。同时,以梯度引导动作生成替代传统标签反转,在数据流形内搜索最坏情况动作。在D4RL基准上的实验证明,本方法显著优于现有基线,在仅5%污染率下即可高效攻破代表性安全约束算法,同时保持干净环境下的性能不变。

原文摘要 · Abstract (English)

Offline Reinforcement Learning (RL) enables policy optimization from static datasets but is inherently vulnerable to backdoor attacks. Existing attack strategies typically struggle against safety-constrained algorithms (e.g., CQL) due to inefficient random poisoning and the use of easily detectable Out-of-Distribution (OOD) triggers. In this paper, we propose CS-GBA (Critical Sample-based Gradient-guided Backdoor Attack), a novel framework designed to achieve high stealthiness and destructiveness under a strict budget. Leveraging the theoretical insight that samples with high Temporal Difference (TD) errors are pivotal for value function convergence, we introduce an adaptive Critical Sample Selection strategy that concentrates the attack budget on the most influential transitions. To evade OOD detection, we propose a Correlation-Breaking Trigger mechanism that exploits the physical mutual exclusivity of state features (e.g., 95th percentile boundaries) to remain statistically concealed. Furthermore, we replace the conventional label inversion with a Gradient-Guided Action Generation mechanism, which searches for worst-case actions within the data manifold using the victim Q-network's gradient. Empirical results on D4RL benchmarks demonstrate that our method significantly outperforms state-of-the-art baselines, achieving high attack success rates against representative safety-constrained algorithms with a minimal 5% poisoning budget, while maintaining the agent's performance in clean environments.

后门攻击离线强化学习安全性对抗样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。