arXiv:2606.16242cs.LGcs.CL2026-06

揭露快速响应框架的中毒漏洞,可极低比例污染训练数据导致检测失效。

Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework

论文配图:Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework
图 1 · 摘自论文原文
  • 利用提示注入将恶意样本植入训练集,实现精准误导。
  • 仅1%污染率即引发近100%误报或96%漏报,破坏检测能力。
  • 适用于研究安全防御机制的人员,揭示生产级系统的隐性风险。

快速响应(RR)框架在生产系统中持续改进越狱检测分类器,通过生成合成变体来应对新出现的越狱攻击。本文揭示,提示注入可渗透该流程,将中毒样本引入训练集,实现两类攻击:(I) 针对性中毒攻击,使无害样本被错误标记为越狱,特定特征(如格式、主题或关键词)触发;(II) 基于概念的后门攻击,使越狱输入在触发器存在时产生误判,甚至针对防御者已训练过的攻击策略也有效。攻击者仅能修改越狱样本(不可篡改良性数据或标签),这一限制此前未被探索,挑战极大。我们提出Omission Attack,利用新发现现象:当训练数据中缺乏某概念的不安全样本时,分类器会错误关联该概念的存在与安全标签。两类攻击均在仅1%污染率下实现显著标签翻转,最高达100%误报率和96%误报率。

原文摘要 · Abstract (English)

The Rapid Response (RR) framework, deployed in production systems, including Anthropic's ASL-3 safeguards, continuously improves jailbreak-detection classifiers. When new jailbreaks emerge that bypass these classifiers, Rapid Response generates synthetic variants for training, helping the model generalize from the new attacks and quickly adapt. We reveal that prompt injection can infiltrate this pipeline to deliver poisoned samples into the classifier's training set, enabling two attack objectives: (I) targeted poisoning attacks that create false positives on harmless samples by categorizing them as a jailbreak, with a specific desired feature (e.g., certain formatting, subject, or keyword), (II) concept-based backdoor attacks that induce false negatives on jailbreak inputs, generalizing even to jailbreaks from attack strategies the defender explicitly trained against, when the backdoor trigger is present. Importantly, our threat model restricts adversaries to modifying only jailbreak samples (not benign data or labels), a constraint unexplored by prior work that makes the second objective particularly challenging. We address this with Omission Attack, which exploits a new phenomenon: when training on concept-absent unsafe samples, the classifier misassociates that concept's presence with the safe label. Both attacks cause substantial and in some cases near-complete label flipping at only a 1% poisoning rate, achieving up to 100% false positive rates and up to 96% false negative rates.

模型安全中毒攻击越狱检测后门攻击

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。