测试代码智能体在语义不变改写下的可靠性,发现顶尖模型表现不一。
A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

- 用语义等价变换生成代码变体,评估智能体修复能力变化
- 多数配置下修复率下降最多6.7个百分点,6组配置显著退化
- 不同框架下模型鲁棒性排名迥异,简单框架更稳定
AI代码智能体日益用于解决实际软件问题,但其在表面代码变化下的可靠性尚不明确。本文评估当代码库被重写为语义等价形式时,修复项目级问题的智能体是否仍可靠。提出随机变体采样器,通过控制流重写、无用代码注入和标识符重命名等常见语义保持变换(SPTs)生成扰动变体。在SWE-bench Verified与SWE-bench Pro数据集上,对两个智能体框架(mini-SWE agent与OpenCode)分别搭配四个前沿模型(Claude Opus 4.5、Kimi K2.5、MiniMax M2.5、Qwen 3.6-27B)进行测试。每例在原始与扰动变体上多次运行,通过配对修复率估计分离扰动影响。结果显示多数配置修复率小幅下降,最差情况平均下降6.7个百分点,16组配置中有6组统计显著退化。关键发现:各模型在不同框架下的鲁棒性排序不一致——例如Qwen在mini-SWE agent上最稳健,但在OpenCode下最脆弱,揭示出复杂的鲁棒性边界。简单框架(mini-SWE agent)更具鲁棒性。结果表明,即使顶级模型也易受语义保持扰动影响,且效果非均匀,提示其在多样化真实代码库中部署时存在可靠性风险。
原文摘要 · Abstract (English)
AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。