研究大模型对不同渠道恶意指令的信任差异,发现工具元数据更易被忽略。
Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models
- 用相同恶意内容在用户消息、工具元数据、工具输出中测试模型反应
- 通用模型对工具元数据指令信任度下降超50%,而代理模型几乎不受影响
- 工具描述比输出更像指令,适合关注安全漏洞的研究者阅读
随着语言模型承担调用API、读取工具输出并操作第三方内容的代理角色,其攻击面已超出用户输入范围。恶意指令通过不同渠道(用户消息、工具元数据、工具输出)到达时,模型是否同等对待尚无系统研究。本文提出安全不对称分数(SAS),通过保持恶意文本一致、仅改变传递渠道的匹配样本对,衡量模型对恶意内容的敏感度变化。在10个上线大模型和3类攻击中,通用模型对工具元数据中的指令显著降低信任(降幅超50%),而代理原生模型则几乎不受影响。该差异在控制工具可用性与评分后依然存在,且经大小匹配的混合效应分析确认。在工具表面内部,相同内容在工具描述中表现得更像指令,尽管功能接口一致。模型将工具元数据视为指令,工具结果视为数据。在Llama 3.3-70B和GPT-OSS-120B中,此信号在中后期层以非线性方式编码,线性探测无法捕捉,但激活修补可恢复。
原文摘要 · Abstract (English)
As language models take on agentic roles that call APIs, read tool outputs, and act on third-party content, their attack surface expands beyond what users type. Whether they treat a malicious instruction the same way regardless of where it arrives has not been studied systematically. We introduce the Safety Asymmetry Score (SAS), measuring how a model's susceptibility to adversarial content shifts depending on whether it arrives in the user message, tool metadata, or tool output, using matched payload pairs that hold the malicious text identical and vary only the channel. Across 10 production LLMs and three attack families, general-purpose models sharply discount instructions arriving as tool metadata relative to identical instructions in the user message, while agent-native models discount them far less. This differential survives an affordance-matched control equalizing tool availability and scoring, and a size-controlled mixed-effects analysis. Within the tool surface the pattern reverses: the same content is far more instruction-like in a tool's description than in its output, with affordances identical. Models treat tool metadata as instructions and tool results as data. In Llama 3.3-70B and GPT-OSS-120B this signal is causally present at mid-to-late depths but non-linearly encoded, so linear probes miss what activation patching recovers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。