首个针对LLVM编译器问题修复的大型基准测试，提升AI修复效率。
LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
- 构建首个真实世界LLVM问题基准集，含423个验证任务。
- 提出自动化评估平台LLVM-Gym，支持自动复现与测试。
- 设计轻量级集成方法LLVM-Ens，提升修复率至21.99%。
LLVM是广泛应用的编译器基础设施，其规模和复杂性导致问题修复耗时费力。尽管大语言模型（LLMs）在问题修复方面取得显著进展，但其在复杂系统级LLVM编译器上的表现仍缺乏研究。为此，我们提出LLVM-Bench，首个面向LLVM问题修复的大规模基准测试，包含423个从LLVM项目中收集并验证的真实任务。我们进一步开发了可扩展的评估平台LLVM-Gym，实现问题复现、补丁应用、编译构建和测试执行的自动化。基于该平台，我们对四种代表性LLM、六种检索配置和三种智能体进行了全面评估。结果表明，当前基于LLM的问题修复技术在LLVM-Bench上仍受限，主要失败模式为补丁无效和构建失败。我们还发现不同LLM与智能体间具有显著互补性，由此提出轻量级集成方法LLVM-Ens：通过整合多种技术生成的补丁，过滤错误冗余候选，识别最优解。实验显示，LLVM-Ens将修复率提升至21.99%，显著推动了LLVM问题修复的进展。
原文摘要 · Abstract (English)
LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。