用大模型自动分类软件问题,少数据也能高准确率。
Applying Large Language Models to Issue Classification: Revisiting with Extended Data and New Models
- 选GPT-4o和DeepSeek R1做分类,用少量数据微调。
- GPT-4o在小数据集上F1达80.7%,比DeepSeek高20%。
- 数据量增大不提升效果,减少对海量训练数据依赖。
软件工程中有效优先处理问题报告有助于优化资源分配与信息恢复。然而,人工分类费时且难以扩展。尽管开源项目已有自动化流程,但通常依赖大规模数据进行训练。传统方法使用机器学习,近年大语言模型(LLMs)在代码生成、需求映射、代码审查等任务中表现突出。本研究探索基于LLM的自动化问题分类方法,旨在降低对大规模训练数据的依赖,同时保持分类可靠性。我们选取两个主流大模型进行对比实验,在多个数据集上评估其性能。结果表明,GPT-4o在NLBSE 2024竞赛数据集上表现最佳;在相同数据集(为NLBSE 2023数据集的十倍大小)上,其F1得分比DeepSeek R1高出20%。微调后GPT-4o平均F1为80.7%,而DeepSeek R1为59.33%。增加数据量未进一步提升性能,说明该方法可减少对海量数据的依赖,实现高效问题分类。
原文摘要 · Abstract (English)
Effective prioritization of issue reports in software engineering helps to optimize resource allocation and information recovery. However, manual issue classification is laborious and lacks scalability. As an alternative, many open source software (OSS) projects employ automated processes for this task, yet this method often relies on large datasets for adequate training. Traditionally, machine learning techniques have been used for issue classification. More recently, large language models (LLMs) have emerged as powerful tools for addressing a range of software engineering challenges, including code and test generation, mapping new requirements to legacy software endpoints, and conducting code reviews. The following research investigates an automated approach to issue classification based on LLMs. By leveraging the capabilities of such models, we aim to develop a robust system for prioritizing issue reports, mitigating the necessity for extensive training data while also maintaining reliability in classification. In our research, we developed an LLM-based approach for accurately labeling issues by selecting two of the most prominent large language models. We then compared their performance across multiple datasets. Our findings show that GPT-4o achieved the best results in classifying issues from the NLBSE 2024 competition. Moreover, GPT-4o outperformed DeepSeek R1, achieving an F1 score 20% higher when both models were trained on the same dataset from the NLBSE 2023 competition, which was ten times larger than the NLBSE 2024 dataset. The fine-tuned GPT-4o model attained an average F1 score of 80.7%, while the fine-tuned DeepSeek R1 model achieved 59.33%. Increasing the dataset size did not improve the F1 score, reducing the dependence on massive datasets for building an efficient solution to issue classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。