为编译器漏洞修复设计专用智能体工具链，提升大模型修复效率。
Agentic Harness for Real-World Compilers
- 构建面向LLM智能体的编译器专用工具链llvm-harness
- 在334个可复现的中间层漏洞上使模型性能提升62%
- 专用于LLVM中间层修复，适合编译器工程与AI融合研究者
编译器对现代计算至关重要，但修复其漏洞极具挑战性。由于漏洞复杂、跨领域知识要求高且报告信息匮乏，需依赖编译器专用工具链。本文提出首个面向大语言模型（LLM）智能体的编译器辅助框架llvm-harness，聚焦于广泛使用的LLVM中间层。该框架包含友好的智能体工具、包含334个可复现漏洞的基准集llvm-bench，以及专门用于自动修复的mini智能体llvm-autofix-mini。评估五种前沿模型发现，使用现有智能体处理编译器漏洞时性能下降；经llvm-harness增强后，性能提升62%。专用智能体llvm-autofix-mini进一步超越增强后的先进模型22%。结果表明，专用工具链对大模型完成编译器任务至关重要。尽管成效显著，专家评审仍揭示若干待解决问题。
原文摘要 · Abstract (English)
Compilers are critical to modern computing, yet fixing compiler bugs is difficult. While recent large language model (LLM) advancements enable automated bug repair, compiler bugs pose unique challenges due to their complexity, deep cross-domain expertise requirements, and sparse, non-descriptive bug reports, necessitating compiler-specific harnesses. To bridge the gap, we introduce llvm-harness, the first harness designed to assist LLM agents in understanding and fixing compiler bugs. Our current focus is on the middle end of LLVM, one of the most widely used compiler infrastructures. Central to llvm-harness are agent-friendly LLVM tools, a benchmark llvm-bench of 334 reproducible LLVM middle-end bugs, and a tailored mini agent llvm-autofix-mini for fixing LLVM middle-end bugs automatically. We evaluate five frontier models and find that they exhibit a performance decline when tackling compiler bugs with the state-of-the-art agent. With llvm-harness' enhancement, their performance improves by 62%. Our specialized mini agent llvm-autofix-mini further outperforms the llvm-harness-enhanced state-of-the-art by 22%. This emphasizes the necessity for specialized harnesses like ours to assist LLMs in compiler engineering tasks. Despite promising results, our expert review also reveals several open challenges that remain when applying LLMs for compiler engineering tasks. GitHub: https://github.com/dtcxzyw/llvm-harness
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。