用大模型生成代码注释，让编程模型学得更好。
Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
- 用大模型自动生成代码注释替代人工注释
- 新数据集使代码理解任务性能提升显著
- 适合关注代码智能与数据构建的研究者
预训练代码模型严重依赖高质量的预训练数据，尤其是连接代码与自然语言的人工编写参考注释。然而，随着软件演进，这些注释常因过时而降低模型性能。大语言模型（LLMs）在生成高质量代码注释方面表现出色。本文探究用LLM生成的注释替换人工注释是否能改善预训练数据集。由于标准指标无法评估参考注释质量，我们提出两项无参考评价任务：代码-注释不一致检测与语义代码搜索。结果表明，LLM生成的注释在语义上比人工注释更一致，经人工评估验证。基于此发现，我们使用LLM生成注释重建CodeSearchNet数据集，并重新预训练CodeT5。评估显示，使用增强数据训练的模型在代码摘要、生成和翻译任务中均优于使用原始人工注释的模型。本工作验证了利用大模型重建预训练数据集以推进代码智能的有效性，挑战了对人工参考注释的传统依赖。
原文摘要 · Abstract (English)
Pre-trained code models rely heavily on high-quality pre-training data, particularly human-written reference comments that bridge code and natural language. However, these comments often become outdated as software evolves, degrading model performance. Large language models (LLMs) excel at generating high-quality code comments. We investigate whether replacing human-written comments with LLM-generated ones improves pre-training datasets. Since standard metrics cannot assess reference comment quality, we propose two novel reference-free evaluation tasks: code-comment inconsistency detection and semantic code search. Results show that LLM-generated comments are more semantically consistent with code than human-written ones, as confirmed by manual evaluation. Leveraging this finding, we rebuild the CodeSearchNet dataset with LLM-generated comments and re-pre-train CodeT5. Evaluations demonstrate that models trained on LLM-enhanced data outperform those using original human comments in code summarization, generation, and translation tasks. This work validates rebuilding pre-training datasets with LLMs to advance code intelligence, challenging the traditional reliance on human reference comments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。