arXiv:2608.05430cs.CRcs.LG2026-08被引 1

提出可感知上下文与查询的恶意指令检测方法,有效防御自适应攻击。

Robust Context-Aware Detection of Malicious Instructions in Text

论文配图:Robust Context-Aware Detection of Malicious Instructions in Text
图 1 · 摘自论文原文
  • 基于上下文和查询信息进行细粒度恶意句子分类
  • 在静态攻击下优于现有基线,在自适应攻击中成功率降低60%以上
  • 支持领域自适应调参,适合实际应用部署

现代大模型强大的指令遵循能力使其成为自主代理的核心,但同时也易受间接提示注入(IPI)等攻击。本文针对文本中恶意指令的精准分割难题,提出一种兼具上下文与查询感知的恶意句子分类方法。为增强对自适应规避攻击的鲁棒性,设计两种对抗训练策略:一是基于嵌入空间的投影梯度优化模拟攻击;二是通过大模型改写生成真实可实现的规避样本。两个方法均支持效用与鲁棒性的平滑权衡。在多个IPI基准测试中,该方法在静态攻击下超越现有最优基线,面对自适应攻击时,攻击成功率显著下降(最高达60%以上),同时保持更高任务可用性。实验还表明最佳参数依赖具体应用领域,因此实际部署需进行领域调优。代码已开源。

原文摘要 · Abstract (English)

The remarkable instruction-following ability of modern LLMs has enabled their practical use as the minds of agents that can autonomously complete increasingly complex tasks. Therein, however, also lies their vulnerability to attacks which embed malicious instructions in text, common variants of which are known as indirect prompt injection (IPI). A fundamental task in addressing this vulnerability is successful segmentation of a given text into benign and malicious sentences (if any). While a number of approaches for this task have been proposed, no detector combines query-relative detection at the segment level, and none are hardened against adaptive evasion attacks realizable in agentic executions. We address the former limitation by developing an approach for malicious sentence classification that is both context- and query-aware. Next, to harden the resulting classifier against evasion, we present two adversarial training methods. The first is directly adapted feature-space adversarial training (AT) in which evasions are approximated using projected-gradient-based optimization in the embedding space. The second simulates realizable evasion attacks in the AT loop through LLM-based paraphrasing. Crucially, we parametrize both AT variants to facilitate a smooth tradeoff between utility and attack robustness. In extensive experiments using indirect prompt injection benchmarks we show that the proposed approach outperforms state-of-the-art IPI defense baselines under static attacks, while in the case of adaptive attacks, our AT variants provide significantly higher utility, lower attack success rate, and often both. Finally, we show that the best AT parameters can depend intimately on the particular application domain. Consequently, domain-dependent tuning of malicious text detectors is likely necessary in practice. Our code is publicly available at https://github.com/tavia-liu/CAD.

恶意检测大模型安全对抗训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。