通过分析模型激活轨迹,检测多轮提示注入攻击的隐蔽信号。
Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection

- 从模型残差流中提取攻击路径的激活轨迹特征
- 在合成数据上将检测率从76.2%提升至93.8%
- 需使用特定来源数据训练,适合安全防护研究者
多轮提示注入攻击通常遵循建立信任、转移目标、升级威胁的路径,但文本层面的防御难以发现每一轮都看似无害的隐蔽攻击。我们发现该攻击路径会在模型残差流中留下激活级别的特征:每个阶段转换都会引起激活位置移动,导致总路径长度远超正常对话。我们称之为对抗性躁动。五个标量轨迹特征可捕捉此信号,使对话级检测准确率在合成保留数据上从76.2%提升至93.8%。该信号在四个模型族(24B-70B)中复现;探测器为模型特异性,不跨架构迁移。泛化能力依赖于训练数据来源:留一源评估显示,合成数据、LMSYS-Chat-1M和SafeDialBench分别捕获不同攻击分布,当真实世界LMSYS攻击分布出现在训练中时,检测率可达47%-71%。三源联合训练在混合保留集上实现89.4%检测率,误报率仅2.4%。进一步表明,我们合成数据中独有的三阶段回合标签(良性/转移/对抗)至关重要:仅用二值对话标签会导致50%-59%误报。结果确立了对抗性躁动作为可靠激活级信号,并明确了实际部署所需的数据条件。
原文摘要 · Abstract (English)
Multi-turn prompt injection follows a known attack path -- trust-building, pivoting, escalation but text-level defenses miss covert attacks where individual turns appear benign. We show this attack path leaves an activation-level signature in the model's residual stream: each phase shift moves the activation, producing a total path length far exceeding benign conversations. We call this adversarial restlessness. Five scalar trajectory features capturing this signal lift conversation-level detection from 76.2% to 93.8% on synthetic held-out data. The signal replicates across four model families (24B-70B); probes are model-specific and do not transfer across architectures. Generalization is source-dependent: leave-one-source-out evaluation shows each of synthetic, LMSYS-Chat-1M, and SafeDialBench captures distinct attack distributions, with detection on real-world LMSYS reaching 47-71% when its distribution is represented in training. Combined three-source training achieves 89.4% detection at 2.4% false positive rate on a held-out mixed set. We further show that three-phase turn-level labels(benign/pivoting/adversarial) unique to our synthetic dataset are essential: binary conversation-level labels produce 50-59% false positives. These results establish adversarial restlessness as a reliable activation-level signal and characterize the data requirements for practical deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。