arXiv:2606.08960cs.CRcs.AI2026-06被引 2

用黑客-修复者循环自动强化智能体评测基准,防住16%的奖励漏洞。

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

  • 引入黑客-修复者循环,让大模型自动发现并修补评测验证器漏洞。
  • 在KernelBench上将攻击成功率从62%降至0%,跨任务修复有效。
  • 弱模型也能防御强黑客,适合评测系统开发者和安全研究者使用。

智能体评测基准依赖人工编写的结局验证器,易受奖励劫持攻击。我们审计了五个终端智能体基准中的1968个任务,发现323个(16%)可被前沿模型仅凭任务描述攻破。这污染了排行榜和强化学习训练信号,而传统应对方式为手动且被动。本文提出黑客-修复者循环:由三个大模型代理交替执行——黑客尝试绕过任务真实求解通过验证器,修复者针对发现的漏洞修补验证器,求解者确认修复后仍允许合法解。循环迭代中,每次修补重塑奖励信号,暴露新漏洞。进一步引入验证器访问与跨任务迁移机制,扩大漏洞探测范围。在KernelBench上,该循环将公开报告漏洞集的攻击成功率从62%降至0%。更关键的是,较弱代理如Gemini 3 Flash的循环可使更强的Gemini 3.1 Pro与Claude Opus 4.7攻击成功率分别从76%、61%降至0%,在Terminal Bench上也使同级别模型攻击率从39%降至17%(覆盖77个任务)。我们发布Terminal Wrench数据集,包含323个可被攻破环境、3632条攻击轨迹、所有发现的漏洞及修复后的验证器,供后续研究复用。

原文摘要 · Abstract (English)

Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking. We audit 1,968 tasks across five terminal-agent benchmarks and find 323 (16%) hackable by frontier models given only the task description. This corrupts both leaderboard rankings and RL training signal, yet the standard response is manual and reactive. We introduce the hacker-fixer loop, a method for building exploit-resistant verifiers without per-task manual patching. The loop alternates three LLM agents: a hacker tries to pass the verifier without solving the task, a fixer patches the verifier to reject each discovered exploit, and a solver confirms the patched verifier still admits legitimate solutions. The loop iterates: each patch reshapes what the verifier rewards, surfacing the next exploit. We further add verifier access, and let patches transfer across tasks, to broaden the exploits the loop discovers. On KernelBench, the loop drives the attack success rate from 62% to 0% on a held-out corpus of publicly reported exploits. We also find that weaker agents in the loop can defend against much stronger hackers: Gemini 3 Flash's loop drives the stronger Gemini 3.1 Pro and Claude Opus 4.7's attack success rate from 76% and 61% to 0% on KernelBench, and Gemini 3.1 Pro's from 39% to 17% on Terminal Bench across 77 tasks. We release Terminal Wrench (323 hackable environments, 3,632 hack trajectories) as a snapshot of the current attack surface, our patched verifiers, the exploits the loop discovered, and our implementation as a basis for future work.

智能体评测奖励劫持对抗训练LLM安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。