arXiv:2605.03952cs.CRcs.AI2026-05被引 4

测试编码模型在分步任务中如何被诱导生成漏洞代码。

MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents

论文配图:MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
图 1 · 摘自论文原文
  • 设计199个三阶段攻击链,模拟看似无害的任务逐步导致恶意结果。
  • 九个主流编码代理在分步任务中漏洞生成率达53%-86%,仅两次拒绝。
  • 将评审者视为对抗测试员可显著提升检测率,误报率仅4.6%。

编码代理虽能通过单次提示的安全审查,但在任务被拆分为常规工程工单时仍会生成可被利用的代码。问题本质在于:现有安全对齐只评估孤立的显式请求,无法识别由一系列看似无害请求序列执行后产生的恶意最终状态。我们提出MOSAIC-Bench(恶意目标以无害合规形式呈现),包含199个三阶段攻击链,搭配部署软件底座上的确定性漏洞验证器(10个Web应用底座、31类CWE漏洞、5种编程语言),将漏洞真实情况和下游评审流程均作为核心评估维度。在该基准上,来自Anthropic、OpenAI、Google、Moonshot、Zhipu、Minimax的九个生产级编码代理在分步任务中漏洞生成率高达53%-86%,全程仅2次拒绝。在相同条件下直接提示实验中,漏洞输出率降至0-20.4%:Claude主要拒绝,Codex则主动加固而非输出漏洞——说明分步任务使两种防御机制同时失效。下游评审代理批准了25.8%的已确认漏洞合并请求;完整上下文实现协议仅缩小50%的差距,排除了上下文碎片化是唯一原因。作为可部署但非自适应的缓解方案,将评审者设定为对抗渗透测试员,可显著降低逃逸率(3.0%-17.6%);在该框架下,开源的Gemma-4-E4B-it评审器在608个真实GitHub PR上检测到88.4%的攻击,误报率仅为4.6%。

原文摘要 · Abstract (English)

Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets. The challenge is structural: existing safety alignment evaluates overt requests in isolation, leaving models blind to malicious end-states that emerge from sequenced compliance with innocuous-looking requests. We introduce MOSAIC-Bench (Malicious Objectives Sequenced As Innocuous Compliance), a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates (10 web-application substrates, 31 CWE classes, 5 programming languages) that treats both exploit ground truth and downstream reviewer protocol as first-class evaluation axes. On this benchmark, nine production coding agents from Anthropic, OpenAI, Google, Moonshot, Zhipu, and Minimax compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs. In a matched direct-prompt experiment over four frontier Claude/Codex agents, vulnerable-output rates fall to 0-20.4%: Claude primarily refuses, while Codex primarily hardens rather than emitting the vulnerable implementation - ticket staging silences both defense modes simultaneously. Downstream, code reviewer agents approve 25.8% of these confirmed-vulnerable cumulative diffs as routine PRs, and a full-context implementation protocol closes only 50% of the staged/direct gap, ruling out context fragmentation as the sole explanation. As a deployable but non-adaptive mitigation, reframing the reviewer as an adversarial pentester reduces evasion across the evaluated reviewer subset; pentester framed evasion ranges from 3.0% to 17.6%, and an open-weight Gemma-4-E4B-it reviewer under this framing detects 88.4% of attacks on the dataset with a 4.6% false-positive rate measured on 608 real-world GitHub PRs.

编码安全漏洞诱导评测基准对抗测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。