首个专用于检测代码提交消息与变更不一致的基准测试,提升代码审查质量。
CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection
- 构建包含7类不一致消息的标注数据集,验证正负样本可靠性。
- 大模型平均召回率85.95%,但语义层级不一致检测仍存短板。
- 少样本提示和思维链可提效,但增加计算开销或误判风险。
版本控制依赖提交消息解释代码变更原因,但消息常质量低且与实际代码变动不一致——即消息-代码不一致(MCI)。MCI误导评审、妨碍维护、污染研究数据,甚至掩盖安全补丁。目前尚无专门用于评估大语言模型(LLMs)MCI检测能力的基准。本文提出CODEFUSE-COMMITEVAL,首个面向该任务的基准。基于ApacheCM数据集构建多样高质量样本,通过规则引导变异生成七类不一致消息,并采用双重复核验证正负样本。利用该标注的文本-差异对数据集,我们在零样本设置及三种增强策略(少样本提示、思维链、扩展上下文)下评估六种开源LLM。结果表明:模型对不一致提交的检测优于一致提交(平均召回率85.95%,精确率80.28%,特异性63.8%);gpt-oss-20B表现最佳,但推理耗时超其他模型两倍。增强策略效果各异:邻近上下文利于大模型但为小模型引入噪声;少样本提示提升准确率并降低耗能,但普遍提高错误预测;思维链提升精确率与特异性,代价是召回下降和更高令牌消耗。按类型分析显示:组件、文件路径和操作级不一致更易检测,而意图级‘目的’不一致检测准确率低、耗能高。本基准为衡量、比较与推进MCI检测提供了严谨基础,凸显丰富上下文与平衡数据对捕捉高层语义鸿沟的必要性。
原文摘要 · Abstract (English)
Version control relies on commit messages to convey the rationale for code changes, but these messages are often low quality and, more critically, inconsistent with their diffs-known as message-code inconsistency (MCI). MCIs mislead reviewers, hinder maintenance, contaminate research datasets, and may obscure security patches. Yet, no dedicated benchmark exists to evaluate models for MCI detection. We introduce CODEFUSE-COMMITEVAL, the first benchmark designed for MCI detection using large language models (LLMs). Built on the ApacheCM dataset for diversity and quality, we generate seven types of inconsistent messages through rule-guided mutations of originally consistent commits and apply two-fold validation to verify both positive and negative samples. Using this labeled dataset of message-diff pairs, we evaluate six state-of-the-art open-source LLMs under a vanilla setting and with three augmentation strategies: few-shot prompting, chain-of-thought, and extended context. Results show models detect inconsistent commits more reliably than consistent ones (average Recall 85.95%, Precision 80.28%, Specificity 63.8%); gpt-oss-20B performs best overall but uses over twice the tokens of others. Augmentation effects vary: adjacent context helps larger models but adds noise for smaller ones; few-shot improves accuracy and reduces token use, yet increases universally incorrect predictions; chain-of-thought boosts precision and specificity at the cost of recall and higher token consumption. Type-wise analysis reveals higher detectability for component, file-path, and operation inconsistencies, but lower accuracy and higher token cost for intent-level "purpose" inconsistencies. CODEFUSE-COMMITEVAL provides a rigorous foundation for measuring, comparing, and advancing MCI detection, highlighting the need for richer context and balanced data to capture high-level semantic gaps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。