检测器在攻击变化时误判仍自信,暴露严重盲区。
Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift
- 用固定阈值测试三款检测器在五种攻击变化下的表现
- 漏检攻击时置信度高达0.99~1.00,但漏报率达0.01~0.97
- 内容键控是根本原因,适合安全评估与模型审计者阅读
提示注入检测器作为防护机制,根据输入得分决定信任或拦截。本文研究当攻击分布偏离原始基准时,检测器得分的置信度问题。评估了ProtectAI-v2和两个Prompt-Guard-2检查点,在五个攻击偏移下使用单一固定校准阈值。报告了严重性度量S:对漏检攻击的置信度,同时给出假阴性率与区分能力。所有检测器在各偏移下,漏检攻击的置信度均在0.99至1.00之间,而假阴性率从0.01到0.97不等——即漏检时近乎确信。三者均未能识别间接行为劫持注入,此盲点在两家厂商、四种规模模型中普遍存在。标准合并校准误差未反映此问题,一检测器被评0.06,实际在攻击上校准误差达0.91。在真实模型中运行发现,漏检注入泄露多数有效攻击,通过率与捕捉其他攻击一致。控制实验表明,根源在于内容键控而非注入结构;指令微调的裁判模型同样存在劫持盲点;黑盒重写器利用内容键控制造高置信度漏检,尤其在最危险攻击类别中最有效。代码与数据已公开。
原文摘要 · Abstract (English)
Prompt-injection detectors are deployed as guards: a model scores an input and a downstream system trusts or blocks it on that score. I study the confidence of these scores, not only their accuracy, when the attack distribution shifts away from the clean benchmark on which the operating point was chosen. I evaluate three released detectors, ProtectAI-v2 and two Prompt-Guard-2 checkpoints, at a single source-calibrated threshold that I freeze and transport across five shifts. I report a severity metric S, how confident a detector is on the attacks it misses, alongside the false-negative rate and discrimination. Across every shift and every detector, severity on the missed attacks stays between 0.99 and 1.00 while the false-negative rate ranges from 0.01 to 0.97: when these detectors miss, they miss with near-certainty. All three confidently pass indirect behavior-hijack injection, a blind spot unanimous across two vendors and a fourfold size range. Standard pooled calibration error does not register this; one detector it rates well-calibrated, at 0.06, is miscalibrated at 0.91 on the attacks alone. Run against live models, the missed injections leak the majority of working exploits, passing them at the rate they catch others. A controlled experiment traces the cause to content-keying rather than injection structure, an instruction-tuned model used as a judge shows the same hijack blind spot, and a black-box rewriter exploits the content-keying to manufacture working confident misses, most effectively on the most dangerous attack category. Code and data are public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。