测试大模型代码审查员如何被话术欺骗，发现其漏洞检测能力严重不足。
SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents
- 用历史漏洞修复反推原始漏洞代码，搭配15种社交工程话术构造攻击性提交请求。
- 8个审查模型在约1000个对抗性PR中平均批准率超30%，暴露安全短板。
- 适合关注AI代码审查安全性的开发者与研究者，揭示人机协作中的信任风险。
大型语言模型（LLMs）正被广泛应用于自动化代码审查系统，其审批决定代码是否合并至共享仓库。然而，当攻击者同时控制代码变更和用于掩盖漏洞的说服性拉取请求（PR）文稿时，审查代理能否识别引入漏洞的代码尚不明确。本文提出SEVRA-BENCH（审查代理中漏洞的社会工程化测试基准），衡量审查代理批准此类对抗性PR的频率。每个PR基于历史漏洞修复提交反推得出原始漏洞代码，并以15种社会工程话术包装。这些话术涵盖支持证据、紧迫感、先前批准信号及权威呼吁等维度。基准测试从MITRE 2025年最危险软件弱点前10名项目中提取约1000个对抗性PR进行评估。对8个审查代理的测试显示，它们极易受话术操纵，暴露出显著的安全能力缺口。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed in automated code-review systems, where their approvals can determine which code is merged into shared repositories. However, it is unclear whether review agents can detect vulnerability-introducing code when an attacker controls both the code change and the persuasive Pull Request (PR) narrative designed to mask it. We introduce SEVRA-BENCH (Social Engineering of Vulnerabilities in Review Agents), a benchmark that measures how often a review agent approves such adversarial PR s. Each PR in SEVRA-BENCH is built from a historical commit that fixed a vulnerability. We automatically reverse that fix to extract the original vulnerable code, and submit the resulting code change as a PR wrapped in one of 15 social-engineering framings. To test review-agent resilience to narrative manipulation, these framings vary dimensions such as supporting evidence, conveyed urgency, signals of prior approval, and appeals to authority. SEVRA-BENCH evaluates a retained challenge split of roughly 1000 adversarial PRs drawn from publicly disclosed vulnerability fixes across the top 10 entries of the MITRE's 2025 most dangerous software weaknesses. Evaluating 8 review agents against this benchmark, we reveal that review agents are susceptible to narrative manipulation, exposing a significant gap in security capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。