arXiv:2603.27333cs.SEcs.AI2026-03被引 1

首个面向C/C++编译错误修复的仓库级真实世界基准,解决现有评估不靠谱问题

ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair

  • 从海量GitHub CI记录自动挖掘真实编译失败案例,构建高保真修复数据集
  • 测试12个大模型发现:73%修复语法正确但仅41%语义有效,模型对错误类型有不同专长
  • 适合研究自动化修复、AI编程工具或软件工程可靠性的开发者和研究者

编译错误是软件开发中的普遍且严重挑战,显著影响生产力。为此提出了自动化编译错误修复(ACER)技术。尽管近年有所进展,但其在真实场景下的表现仍缺乏有效评估,主要受限于现有基准的不足:如去上下文的单文件数据、缺乏真实源码多样性,以及忽略仓库级复杂性的局部任务建模。为填补这一关键空白,我们提出ComBench,首个面向C/C++编译错误修复的仓库级、可复现的真实世界基准。ComBench通过新型自动化框架,系统性地从大规模开源项目GitHub CI历史中挖掘真实失败案例。该框架包含高精度识别真实修复补丁的技术,以及高保真还原原始瞬时构建环境的机制。所有样本均经执行验证,确保失败与修复可复现。基于ComBench,我们对12个现代大模型在直接与代理式修复设置下进行了全面评估。实验显示,模型在语法正确性上表现较好(如GPT-5达73%成功),但在语义正确性上较差(仅41%修复补丁有效)。同时发现不同模型对不同类型错误存在差异化专长。ComBench为未来能应对现代软件开发复杂性的ACER技术发展提供了坚实可靠的评估平台。

原文摘要 · Abstract (English)

Compilation errors pose pervasive and critical challenges in software development, significantly hindering productivity. Therefore, Automated Compilation Error Repair (ACER) techniques are proposed to mitigate these issues. Despite recent advancements in ACER, its real-world performance remains poorly evaluated. This can be largely attributed to the limitations of existing benchmarks, \ie decontextualized single-file data, lack of authentic source diversity, and biased local task modeling that ignores crucial repository-level complexities. To bridge this critical gap, we propose ComBench, the first repository-level, reproducible real-world benchmark for C/C++ compilation error repair. ComBench is constructed through a novel, automated framework that systematically mines real-world failures from the GitHub CI histories of large-scale open-source projects. Our framework contributes techniques for the high-precision identification of ground-truth repair patches from complex version histories and a high-fidelity mechanism for reproducing the original, ephemeral build environments. To ensure data quality, all samples in ComBench are execution-verified -- guaranteeing reproducible failures and build success with ground-truth patches. Using ComBench, we conduct a comprehensive evaluation of 12 modern LLMs under both direct and agent-based repair settings. Our experiments reveal a significant gap between a model's ability to achieve syntactic correctness (a 73% success rate for GPT-5) and its ability to ensure semantic correctness (only 41% of its patches are valid). We also find that different models exhibit distinct specializations for different error types. ComBench provides a robust and realistic platform to guide the future development of ACER techniques capable of addressing the complexities of modern software development.

编译错误自动化修复基准测试LLM应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。