arXiv:2605.19192cs.AIcs.CR2026-05被引 1

让大模型的幻觉变成可审计的安全屏障,防止错误指令导致越权操作。

Hallucination as Exploit: Evidence-Carrying Multimodal Agents

论文配图:Hallucination as Exploit: Evidence-Carrying Multimodal Agents
图 1 · 摘自论文原文
  • 用可信凭证替代模型自说自话,仅允许经验证的指令执行
  • 在1700次攻击测试中零绕过,200个端到端任务无一次违规执行
  • 适合关注AI安全、工具调用可靠性的研究者和开发者

多模态智能体越来越多地根据截图、文档和网页内容调用工具,而错误的感知判断可能使幻觉从回答质量问题演变为权限滥用。本文提出证据携带型多模态智能体(ECA),将自由文本视为不可信证据,将每个工具调用分解为关键动作谓词,通过受控的DOM/OCR/AX验证器获取类型化证书,并使用确定性门控机制仅授权证书所支持的权限。不同于隐藏感知误差,ECA将模型信念转化为验证器、模式和实现层面可审计的残差。在17类典型攻击的红队测试中,四项针对性加固措施均必要;加固后,经典门控绕过率为0/1,700(95%置信上限0.22%)。基于内容生成的证书下,ECA在200个端到端任务中零误执行(95%上限2.67%),浏览器任务中为4.3%。对500个分层任务键的HACR审计显示,原始智能体和提示防御均100%和49.6%进入不安全执行,而ECA未出现。对7,488条GPT-5.4轨迹的证书重放验证了门控正确性,而神经裁判基线仍大量允许不安全行为。核心原则是:模型语言可提议工具使用,但必须由认证谓词授权。

原文摘要 · Abstract (English)

Multimodal agents increasingly choose tool calls from screenshots, documents, and webpages, where a false perceptual claim can turn hallucination from an answer-quality error into an authorization failure. We formalize this failure mode as hallucination-to-action conversion: an unsupported claim supplies the precondition for a privileged action. We propose evidence-carrying multimodal agents (ECA), which treat free-form model text as inadmissible evidence, decompose each tool call into action-critical predicates, obtain typed certificates from constrained DOM/OCR/AX verifiers, and use a deterministic gate to authorize only the privileges those certificates support. Rather than hiding perception error, ECA converts opaque model belief into auditable residuals at the verifier, schema, and implementation levels. Verifier red-teaming across 17 canonical attack categories shows that four targeted hardening steps are each necessary; after hardening, canonical gate bypass is 0/1,700 (Wilson 95% upper bound 0.22%). With content-derived certificates, ECA observes zero unsafe executions on 200 end-to-end tasks (Wilson 95% upper bound 2.67%) and 120 browser tasks (upper bound 4.3%). A HACR audit on 500 stratified task keys shows that unsupported action-critical claims reach unsafe execution for naive agents (100.0%) and prompt-only defenses (49.6%), but not for ECA. Oracle-certificate replay over 7,488 GPT-5.4 traces isolates gate correctness, while neural judge baselines still admit most unsafe actions under the same threat model. The resulting principle is simple: model language may propose tool use, but certified predicates must authorize it.

AI安全工具调用可信推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。