arXiv:2605.30521cs.CL2026-05

用模拟工具调用隔离不可信输入,反而降低模型抗攻击能力。

Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs

  • 将不可信输入包裹在模拟工具调用中作为隔离机制
  • 在七种模型上测试,多数任务攻击成功率反而上升
  • 适合关注模型安全与输入防御的开发者参考

大语言模型常需处理不可信输入,如另一模型的回答或垃圾内容分类任务,在对抗压力下易被操纵。当前主流模型(如OpenAI)按指令层级区分信任度,从系统消息(最可信)到工具结果(最不可信)。本文提出将不可信内容封装为模拟工具调用以实现隔离,通过自动化红队测试,在七个模型和三个LLM作为裁判的任务中验证。结果反直觉:在二分类任务(GSM8K评分)中,工具封装反而显著提升攻击成功率;在标量和成对评估任务中影响较小且模型依赖性强,无模型稳定获益,部分出现指令层级倒置。建议在部署系统中评估此局限性,并长期探索更强的指令层级训练或新的不可信输入处理原语。

原文摘要 · Abstract (English)

Large language models must frequently process untrusted inputs, such as judging an answer from another model or running tasks like spam and harm classifiers while under adversarial pressure. These inputs are often string-formatted directly into a prompt template, leaving systems fragile to manipulation. Current LLM specs from major providers like OpenAI distinguish trustworthiness along an Instruction Hierarchy, from System messages (most trusted) to Tool Results (least trusted). A possible natural mitigation is to wrap untrusted content in a mock tool call as a quarantine. We explore this hypothesis with an automated redteaming search over static attack strings across seven models and three LLM-as-a-Judge tasks. Counter to our hypothesis, tool-wrapping does not broadly improve robustness. On a binary evaluation task (GSM8K grading) it typically increases attack success rates, an apparent inversion of the instruction hierarchy. On scalar and pairwise tasks the effect is smaller and model-dependent, with no tested model reliably helped, and several showing inversion. We recommend evaluating this limitation in deployed systems, and longer-term, pursuing stronger Instruction Hierarchy training or new untrusted-input primitives.

模型安全输入防御鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。