提出新基准与框架，显著提升大模型定位内核缺陷的能力
Taming System Complexity: Demystifying Software Engineering Agents in Diagnosing Linux Kernel Faults
- 构建真实内核漏洞数据集LinuxFLBench，评估大模型故障定位性能
- 现有大模型在文件级定位准确率最高仅41.6%，表现受限
- 提出LinuxFL$^+$框架，零成本提升准确率7.2%-11.2%
Linux内核是众多系统的基础，其缺陷可能影响数十亿用户。故障定位（FL）对保障软件质量至关重要。尽管近期大语言模型（LLM）代理在SWE-bench等基准上表现良好，但其在大规模、低可观测性的内核代码中表现尚不明确。本文构建了基于真实漏洞的LinuxFLBench基准，实证评估了先进LLM代理在内核中的表现。结果发现，现有方法在文件级别最高仅达41.6%的准确率。为此，我们提出LinuxFL$^+$增强框架，显著提升所有被测代理的定位效果（提升7.2%-11.2%），且开销极低。
原文摘要 · Abstract (English)
The Linux kernel is a critical system, serving as the foundation for numerous systems. Bugs in the Linux kernel can cause serious consequences, affecting billions of users. Fault localization (FL), which aims at identifying the buggy code elements in software, plays an essential role in software quality assurance. While recent LLM agents have achieved promising accuracy in FL on recent benchmarks like SWE-bench, it remains unclear how well these methods perform in the Linux kernel, where FL is much more challenging due to the large-scale code base, limited observability, and diverse impact factors. In this paper, we introduce LinuxFLBench, a FL benchmark constructed from real-world Linux kernel bugs. We conduct an empirical study to assess the performance of state-of-the-art LLM agents on the Linux kernel. Our initial results reveal that existing agents struggle with this task, achieving a best top-1 accuracy of only 41.6% at file level. To address this challenge, we propose LinuxFL$^+$, an enhancement framework designed to improve FL effectiveness of LLM agents for the Linux kernel. LinuxFL$^+$ substantially improves the FL accuracy of all studied agents (e.g., 7.2% - 11.2% accuracy increase) with minimal costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。