测试编程智能体在用户实时修改代码时的应对能力，发现其易因忽略代码变更而失败。
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
- 通过模拟用户冲突性修改，构建真实协作场景下的压力测试框架。
- 在基准上平均解决率下降7.7个百分点，长任务中退化依然明显。
- 揭示智能体缺乏对工作区演化的感知，适合研究人机协同编程的学者。
现实软件开发需要编程智能体在用户可能随时查看和修改代码的共享环境中运行，但现有仓库级基准通常让智能体独立工作或仅限制用户通过消息参与。这引发疑问：编程智能体如何理解并响应共享工作区中的代码变化？我们提出SWE-Touch，通过经验证的反向修改（Counter-Edits）来压力测试该场景——即与任务目标冲突的任务相关代码的合理修改。SWE-Touch从多个修复轨迹中挖掘关键代码区域，使用独立的用户补丁生成器构造修改，并在智能体触及相关代码时注入上下文消息。我们在SWE-bench Verified上评估了九个编程模型，额外在更长任务周期的SWE-Bench Pro和DeepSWE上展开实验。反向修改使SWE-bench Verified上的平均解决率下降7.7个百分点，且在长周期基准上退化持续存在。轨迹分析表明这些失败源于对演化工作区的感知不足：智能体可能保留冲突代码或替换后未充分重新检查仓库，也未用针对性测试验证修改行为。结果表明，强自主性能并不等同于共享协作所需的环境感知与自适应能力，未来优化应聚焦检测工作区变化、协调冲突修改与任务目标、以及验证受影响行为的能力。
原文摘要 · Abstract (English)
Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。