用大模型自动分析集成测试失败日志，定位故障根源。
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
- 利用大模型分析日志，提取关键信息并生成摘要。
- 在71个真实案例中诊断准确率达90.14%，52,635次测试使用。
- 已集成到Google代码审查系统，开发者反馈帮助显著。
集成测试对复杂软件系统的质量和可靠性至关重要，但其失败诊断因日志量大、非结构化且异构而面临巨大挑战，导致认知负荷高、信噪比低，诊断耗时且困难。开发者普遍反映集成测试故障诊断耗时远超单元测试。为此，我们提出Auto-Diagnose，一款基于大模型的诊断工具，可分析失败日志，生成包含最相关日志行的简洁摘要，并集成至Google内部代码审查系统Critique，提供上下文感知的实时辅助。案例研究显示，对71个真实故障的手动评估中，诊断准确率达90.14%。在谷歌范围部署后，该工具被应用于52,635个不同的失败测试。用户反馈显示，仅5.8%的案例认为“无帮助”，在370个Post结果的工具中排名第14位。用户访谈确认了其实际价值，以及将自动诊断融入现有工作流的积极接受度。结论表明，大模型凭借处理和总结复杂文本数据的能力，在集成测试故障诊断中表现优异。将此类AI工具自动嵌入开发者日常流程，广受好评，但工具准确性仍是影响采纳和感知价值的关键因素。
原文摘要 · Abstract (English)
Integration testing is critical for the quality and reliability of complex software systems. However, diagnosing their failures presents significant challenges due to the massive volume, unstructured nature, and heterogeneity of logs they generate. These result in a high cognitive load, low signal-to-noise ratio, and make diagnosis difficult and time-consuming. Developers complain about these difficulties consistently and report spending substantially more time diagnosing integration test failures compared to unit test failures. To address these shortcomings, we introduce Auto-Diagnose, a novel diagnosis tool that leverages LLMs to help developers efficiently determine the root cause of integration test failures. Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines, and is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance. Based on our case studies, Auto-Diagnose is highly effective. A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause. Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed "Not helpful" in only 5.8% of cases, and it was ranked #14 in helpfulness among 370 tools that post findings in Critique. Finally, user interviews confirmed the perceived usefulness of Auto-Diagnose and positive reception of integrating automatic diagnostic assistance into existing workflows. We conclude that LLMs are highly successful in diagnosing integration test failures due to their capacity to process and summarize complex textual data. Integrating such AI-powered tooling automatically into developers' daily workflows is perceived positively, with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。