真实交互式编程评测基准，测代码助手协作能力。
SWE-Together: Evaluating Coding Agents in Interactive User Sessions
- 从1.1万次真实会话重构多轮编程任务
- 强模型成功率更高且纠错次数更少
- 适合评估编程助手的交互协作能力
现有编程代理评测多为静态：代理一次性接收完整任务描述，仅以最终代码评判。真实编程辅助是交互式的，用户会不断澄清目标、添加约束并修正错误。我们提出SWE-Together，一个基于真实用户-代理编程会话重建的多轮评测基准。为保证可复现性，我们从11,260次记录会话中筛选出109个仓库级任务，要求具备可恢复的仓库状态、明确的用户目标和可观测结果。为在不同代理间重现这些交互，我们构建了一个基于LLM的反应式用户模拟器，保留原始用户意图，并在代理进展需要时提供反馈。评估时不仅关注最终仓库正确性，还统计交互过程中所需的纠正反馈轮数。对前沿编程代理的实验表明，更强的代理通常能获得更高的最终成功率，同时所需干预更少，表明用户体验更优。
原文摘要 · Abstract (English)
Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。