用提示工程让大模型直接分类代码提交,无需训练数据。
Conventional Commit Classification using Large Language Models and Prompt Engineering

- 通过零样本、少样本和思维链三种提示策略,直接用大模型分析代码差异。
- 少样本提示在3200个提交上达到最高准确率,模型规模越大表现越优。
- 适合不想建标注数据集却想自动分类提交的开发者和研究者。
传统提交消息采用结构化格式,有助于提升可读性、软件维护性,并支持自动化工具(如变更日志生成器和语义版本系统)。现有分类方法通常依赖大规模标注数据训练机器学习/深度学习模型。本文探索了一种无需训练的替代方案:利用大语言模型(LLM)通过提示工程实现分类。我们评估了三种提示策略(零样本、少样本、思维链),在三个不同规模的开源LLM(Mistral-7B-Instruct、LLaMA-3-8B、DeepSeek-R1-32B)上进行测试。分类基于从InfluxDB仓库中挖掘的3200个平衡提交的代码差异,无需任何微调。结果表明,少样本提示始终表现最佳,而思维链提示未带来额外增益。其中,DeepSeek-R1-32B整体性能最强,说明模型规模对分类任务有显著影响。这些发现为研究人员和实践者提供了无需构建和维护标注数据即可自动化提交分类的实用指导。
原文摘要 · Abstract (English)
Conventional commits provide a structured format for writing commit messages, which improves readability, software maintenance, and enables automation tools such as changelog generators and semantic versioning systems. Existing approaches to conventional commit classification typically rely on ML/DL models trained on large labeled datasets. In this paper, we investigated a training-free alternative by leveraging large language models (LLMs) through prompt engineering. Rather than building a task-specific classifier, we evaluate three prompting strategies, such as zero-shot, few-shot, and chain-of-thought, across three open-source LLMs of varying scale: Mistral-7B-Instruct, LLaMA-3-8B, and DeepSeek-R1-32B. Classification is performed directly on code diffs extracted from a balanced dataset of 3,200 commits mined from the InfluxDB repository, without any model fine-tuning. Our results show that few-shot prompting consistently achieves the highest accuracy, while chain-of-thought prompting does not yield additional gains for this classification task. Among the evaluated models, DeepSeek-R1-32B achieves the strongest overall performance, suggesting that model scale plays a meaningful role in conventional commit classification. These findings provide practical guidance for researchers and practitioners seeking to automate commit classification without the overhead of curating and maintaining labeled training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。