arXiv:2607.00481cs.CRcs.AI2026-07

通过模拟审核流程,绕过函数调用大模型的安全限制。

Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

论文配图:Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
图 1 · 摘自论文原文
  • 构建多轮模拟审核轨迹,伪装成合规流程诱导模型输出有害内容。
  • 在五个主流大模型上平均攻击成功率最高,且仅需极少交互次数。
  • 揭示了当前安全防护在函数调用场景下的根本缺陷,适合安全研究者参考。

jailbreak 攻击仍是大语言模型安全部署的关键威胁。以往研究主要关注提示层面的攻防,但本文指出,这种以提示为中心的范式忽略了有状态函数调用环境中的结构性漏洞。在该类应用中,开发者定义的模式、结构化参数和不可信工具输出交织于同一共享上下文,模糊了可信控制逻辑与不可信数据的边界,使恶意意图可分散于多轮执行路径中。我们提出 SMT(Simulated Moderation Traces),一种基于模拟审核轨迹的黑盒攻击框架。不同于纯提示交互,SMT 构建了一条模拟合法审核-审计工作流的多轮轨迹,其中伪造的审核框架以红队测试为借口诱导有害生成。后续验证反馈将安全拒绝视为执行失败,促使模型逐步弱化安全约束,最终触发有害输出。在来自五家不同厂商的主流商业 LLM 上,基于两个标准化安全基准的实证评估显示,SMT 在平均攻击成功率和 HarmScore 上均显著领先现有基线,且查询次数接近最小。结果表明,仅靠提示层净化无法有效防御工具启用的 LLM 系统,亟需对模式、参数、工具输出及累积对话状态进行上下文感知的验证。代码已开源:https://github.com/liujlong27/SMT。

原文摘要 · Abstract (English)

Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation-auditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model's safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.

模型安全对抗攻击函数调用红队测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。