一句话骗过安全检查,让多智能体系统偷偷干违规事
Semantic Intent Fragmentation: A Single-Shot Compositional Attack on Multi-Agent AI Pipelines
- 用一句合法请求诱使系统拆解任务,各环节都看似合规
- 14个企业场景中71%(10/14)生成了违规计划,子任务全通过审查
- 无需注入内容或修改系统,适合研究系统安全漏洞的人看
我们提出语义意图碎片化(SIF),一种针对大模型编排系统的新型攻击。单一合法请求可导致编排器将任务分解为多个看似无害的子任务,但组合后违反安全策略。当前安全机制仅在子任务层面生效,导致每步均通过检测,违规行为仅在整体计划中显现。SIF利用OWASP LLM06:2025漏洞,通过批量范围扩大、静默数据外泄、嵌入触发器部署和准标识符聚合四种机制实现,无需注入内容、系统修改或后续交互。基于OWASP、MITRE ATLAS和NIST框架构建三阶段红队测试管道,覆盖财务报告、信息安全与人力资源分析共14个真实企业场景。使用GPT-20B编排器时,71%(10/14)场景生成政策违规计划,所有子任务均被判定为良性。三种独立信号验证:确定性污染分析、思维链评估及跨模型合规判断器(误报率0%)。更强的编排器反而提升攻击成功率。在计划层面引入信息流追踪与合规评估,可提前100%检测所有攻击,表明组合式安全漏洞可被弥补。
原文摘要 · Abstract (English)
We introduce Semantic Intent Fragmentation (SIF), an attack class against LLM orchestration systems where a single, legitimately phrased request causes an orchestrator to decompose a task into subtasks that are individually benign but jointly violate security policy. Current safety mechanisms operate at the subtask level, so each step clears existing classifiers -- the violation only emerges at the composed plan. SIF exploits OWASP LLM06:2025 through four mechanisms: bulk scope escalation, silent data exfiltration, embedded trigger deployment, and quasi-identifier aggregation, requiring no injected content, no system modification, and no attacker interaction after the initial request. We construct a three-stage red-teaming pipeline grounded in OWASP, MITRE ATLAS, and NIST frameworks to generate realistic enterprise scenarios. Across 14 scenarios spanning financial reporting, information security, and HR analytics, a GPT-20B orchestrator produces policy-violating plans in 71% of cases (10/14) while every subtask appears benign. Three independent signals validate this: deterministic taint analysis, chain-of-thought evaluation, and a cross-model compliance judge with 0% false positives. Stronger orchestrators increase SIF success rates. Plan-level information-flow tracking combined with compliance evaluation detects all attacks before execution, showing the compositional safety gap is closable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。