用大模型自动把混乱的报错信息转成规范报告，提升开发效率。
Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation
- 用指令微调的大模型将非结构化报错转为标准格式报告。
- 微调版Qwen 2.5表现最佳，生成质量达77%（CTQRS）。
- 适合想减少手动整理报错的开发者和维护团队。
缺陷报告包含开发人员定位和修复软件缺陷所需的信息。然而，信息不清晰、不完整或模糊可能导致缺陷分类和解决过程延迟，并增加人工工作量。本文探讨指令微调的大语言模型（LLMs）能否自动将随意、非结构化的缺陷报告转化为高质量、符合标准模板的结构化报告。我们评估了三种开源指令微调模型（Qwen 2.5、Mistral、Llama 3.2）与ChatGPT-4o的表现，采用CTQRS、ROUGE、METEOR和SBERT等指标进行衡量。实验表明，微调后的Qwen 2.5达到77%的CTQRS得分，优于微调后的Mistral（71%）、Llama 3.2（63%）以及3次提示下的ChatGPT（75%）。进一步分析显示，Llama 3.2在识别缺失字段（尤其是预期行为和实际行为）方面准确率更高，而Qwen 2.5在捕获复现步骤上表现更优，F1得分为76%。在其他主流项目（如Eclipse、GCC）上的测试表明，该方法具有良好泛化能力，未见项目中最高达到70%的CTQRS得分。这些发现突显了指令微调在自动化结构化缺陷报告生成中的潜力，有助于降低开发者的手动工作量，优化软件维护流程。
原文摘要 · Abstract (English)
Bug reports contain the information developers need to triage and fix software bugs. However, unclear, incomplete, or ambiguous information may lead to delays and excessive manual effort spent on bug triage and resolution. In this paper, we explore whether Instruction fine-tuned Large Language Models (LLMs) can automatically transform casual, unstructured bug reports into high-quality, structured bug reports adhering to a standard template. We evaluate three open-source instruction-tuned LLMs (\emph{Qwen 2.5, Mistral, and Llama 3.2}) against ChatGPT-4o, measuring performance on established metrics such as CTQRS, ROUGE, METEOR, and SBERT. Our experiments show that fine-tuned Qwen 2.5 achieves a CTQRS score of \textbf{77%}, outperforming both fine-tuned Mistral (\textbf{71%}), Llama 3.2 (\textbf{63%}) and ChatGPT in 3-shot learning (\textbf{75%}). Further analysis reveals that Llama 3.2 shows higher accuracy of detecting missing fields particularly Expected Behavior and Actual Behavior, while Qwen 2.5 demonstrates superior performance in capturing Steps-to-Reproduce, with an F1 score of 76%. Additional testing of the models on other popular projects (e.g., Eclipse, GCC) demonstrates that our approach generalizes well, achieving up to \textbf{70%} CTQRS in unseen projects' bug reports. These findings highlight the potential of instruction fine-tuning in automating structured bug report generation, reducing manual effort for developers and streamlining the software maintenance process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。