SWE-Agent评估只看通过率会误判侥幸过关,新框架揭示了真实开发过程质量差异。
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

- 构建过程级评估框架AgentLens,用上下文意图识别动作阶段
- 在1815条轨迹中发现10.7%为侥幸通过,存在盲目重试等低效行为
- 提供带质量分和错误信号的标注数据集,适合模型优化与评测研究者
SWE-Agent评估长期依赖最终测试通过这一二元信号,导致精心求解与混乱试错被等同对待。我们分析了8个模型后端在60个SWE-bench Verified任务上的2,614条OpenHands轨迹,其中47个任务有足够通过轨迹构建任务级过程参考,形成1,815条轨迹子集。在该子集中,10.7%的通过轨迹表现出‘侥幸通过’(Lucky Pass)特征:回归循环、盲试重试、缺失验证或探索-实现-验证时序错乱。我们提出AgentLens框架,构建AgentLens-Bench数据集,包含1,815条轨迹,标注质量分数、浪费信号、分歧点及47个任务级前缀树接受器(PTA)参考。PTA通过合并同一任务的多个通过解生成,意图标签器基于轨迹历史而非工具身份判断动作类型(探索、实现、验证、编排)。在AgentLens-Bench上,质量分数将通过轨迹划分为侥幸、稳健、理想三类,并将侥幸通过分解为五种常见机制。八个模型后端的侥幸率在0.5%至23.2%之间,部分模型按质量得分排名时上升达五个名次。
原文摘要 · Abstract (English)
Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and define AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We plan to release the project repository soon, including AgentLens-Bench artifacts, the AgentLens SDK, and the analysis tooling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。