arXiv:2605.01078cs.CRcs.AI2026-05

用句子关系检测恶意指令,让大模型更安全。

A Sentence Relation-Based Approach to Sanitizing Malicious Instructions

  • 构建句子关系图,用推理得分识别异常内容
  • 攻击成功率降至接近零,优于九种现有方法
  • 适合依赖外部文本的智能助手和工具型模型

检索增强生成与工具集成的大模型日益依赖外部文本源,这扩大了攻击面,使攻击者可插入恶意指令以触发模型异常行为。现有防御多采用大模型检测器,但易受优化攻击,且训练方法常无法泛化至新数据分布。为此,我们提出SONAR——一种基于句子关系的提示净化框架。该框架在用户查询与外部数据间构建句级关系图,利用蕴含与矛盾得分作为边权重,识别偏离核心任务的句子,再通过连通性驱动剪枝,移除被标记的注入种子及其关联邻居,同时保留良性上下文。跨多个模型与数据集的严格评估显示,SONAR将攻击成功率降至近乎零,显著优于九种基准防御方案。

原文摘要 · Abstract (English)

Retrieval-augmented generation and tool-integrated LLM agents increasingly depend on external textual sources. This reliance broadens the available attack surface, allowing adversaries to insert malicious instructions that trigger unintended model behaviors. Current defensive measures often utilize LLM-based detectors to filter such content, but these approaches remain vulnerable to optimization-based attacks. Additionally, training-based methods frequently fail to generalize to novel data distributions. To resolve these issues, we introduce SONAR, a prompt sanitization framework that identifies and removes injected content using metrics from natural language inference. Specifically, SONAR constructs a sentence-level relational graph across the user query and external data. By using entailment and contradiction scores as edge weights, the system identifies sentences that deviate from the core task. It then employs connectivity-driven pruning to eliminate flagged injection seeds and their related neighbors while maintaining benign context. Rigorous evaluations across several models and datasets show that SONAR reduces the attack success rate to nearly zero, significantly outperforming nine established baseline defenses.

安全大模型防御提示净化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。