arXiv:2608.18280cs.SEcs.AI2026-08

揭示代码问题修复任务难易度的结构根源,可提前预测难度。

What Makes Software Issue Resolution Tasks Difficult for Agents?

论文配图:What Makes Software Issue Resolution Tasks Difficult for Agents?
图 1 · 摘自论文原文
  • 通过分析代码补丁、仓库规模和提示语言特征,量化任务难度
  • 静态特征能准确预测86.3%的任务成功率,碎片化与仓库规模是主因
  • 适合评估智能体能力或构建可控难度基准的研究者使用

随着代理系统在基准测试中迅速饱和,当前评分难以解读,因缺乏对任务难度的控制与刻画。本文提出测量框架,系统量化软件任务结构属性与代理解决缺陷任务成功率之间的关系。我们在迄今最大的开源编码代理轨迹数据集CoderForge-Preview上开展大规模实证研究,提取任务补丁、仓库及提示中的多维特征,并采用集成方法、SHAP归因与效应量分析评估各特征对任务结果的预测能力。结果表明,任务难度可由静态特征高度预测(AU C = 0.863),主要受补丁碎片化程度和仓库规模驱动;提示语言特征在中等难度任务中表现显著,揭示难度的分层结构。结论:任务难度蕴含于其结构之中,支持静态预估,为构建难度可控的评估基准奠定基础。

原文摘要 · Abstract (English)

Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scores remain difficult to interpret due to the lack of control and characterization of task difficulty. More specifically, we currently have little understanding of what makes one task harder than another, and to what extent task difficulty is predictable from static task properties. Aims. We propose a measurement framework to investigate and systematically quantify what structural properties of software tasks correspond to agent success rates for issue resolution tasks. Method. We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date, by extracting features across task patch, repository and prompt. We evaluated the predictive power of each feature against task outcomes using ensemble methods, SHAP attribution, and effect size analysis. Results We found that task difficulty is substantially predictable from static features (AU C = 0.863) and is largely driven by patch fragmentation and repository scale. Prompt linguistic features become visible among top contributors for tasks in the mid-band, revealing a layered structure of difficulty. Conclusion. The difficulty of an issue resolution task is encoded in its structure. This enables static, pre-hoc difficulty estimation and lays the groundwork for difficulty-controlled benchmark construction for evaluation of agents.

智能体评估代码生成难度预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。