用大模型降低代码检测误报率,实测效果显著且成本极低。
Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry
- 结合大模型与静态分析的混合方法识别误报
- 可消除94%-98%误报且保持高召回率
- 每条告警成本仅需2.1-109.5秒,适合工业级应用
静态分析工具(SAT)在学术界和工业界广泛应用以提升软件质量,但其实际使用常受高误报率困扰,尤其在大型企业系统中。这些误报需大量人工排查,造成严重效率损失。尽管已有研究展示大模型在开源基准上减少误报的潜力,其在真实企业环境中的效果仍不明确。为此,我们在腾讯开展首次针对工业场景下多种大模型误报消减技术的综合性实证研究。基于腾讯广告与营销服务系统的定制化静态分析数据,构建包含433个告警的样本集(328个误报,105个真报),涵盖三种常见缺陷类型。通过开发者访谈与数据分析发现,误报普遍,每条告警平均需10-20分钟人工排查。结果表明,大模型结合静态分析的混合方法在工业场景中具有巨大潜力,可消除94%-98%误报并保持高召回率。同时,该方法成本极低,每条告警耗时仅2.1-109.5秒,费用为0.0011-0.12美元,相较人工审查节省数个数量级。案例分析进一步揭示了当前大模型方法在工业应用中的关键局限。
原文摘要 · Abstract (English)
Static analysis tools (SATs) are widely adopted in both academia and industry for improving software quality, yet their practical use is often hindered by high false positive rates, especially in large-scale enterprise systems. These false alarms demand substantial manual inspection, creating severe inefficiencies in industrial code review. While recent work has demonstrated the potential of large language models (LLMs) for false alarm reduction on open-source benchmarks, their effectiveness in real-world enterprise settings remains unclear. To bridge this gap, we conduct the first comprehensive empirical study of diverse LLM-based false alarm reduction techniques in an industrial context at Tencent, one of the largest IT companies in China. Using data from Tencent's enterprise-customized SAT on its large-scale Advertising and Marketing Services software, we construct a dataset of 433 alarms (328 false positives, 105 true positives) covering three common bug types. Through interviewing developers and analyzing the data, our results highlight the prevalence of false positives, which wastes substantial manual effort (e.g., 10-20 minutes of manual inspection per alarm). Meanwhile, our results show the huge potential of LLMs for reducing false alarms in industrial settings (e.g., hybrid techniques of LLM and static analysis eliminate 94-98% of false positives with high recall). Furthermore, LLM-based techniques are cost-effective, with per-alarm costs as low as 2.1-109.5 seconds and $0.0011-$0.12, representing orders-of-magnitude savings compared to manual review. Finally, our case analysis further identifies key limitations of LLM-based false alarm reduction in industrial settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。