GPT-4生成的代码注释对新手程序员与专家水平相当。
Evaluating the Quality of Code Comments Generated by Large Language Models for Novice Programmers
- 用LeetCode简单级Java题目的代码，对比LLM与专家注释质量。
- GPT-4在清晰度、易懂性、概念解释上接近专家水平。
- 相比其他模型，GPT-4更受新手欢迎，尤其在复杂度说明上优势明显。
大型语言模型（LLMs）在为新手程序员生成代码注释方面展现出潜力，但其教学有效性尚未充分评估。本研究评估了GPT-4、GPT-3.5-Turbo和Llama2生成的注释质量，与专家编写注释进行比较，重点关注对初学者的适用性。分析来自LeetCode的“easy”级别Java解法数据集发现，GPT-4在对新手至关重要的方面——如清晰度、新手友好性、概念阐明和分步指导——表现与专家注释相当。GPT-4在讨论复杂度方面优于Llama2（卡方值=11.40，p=0.001），且在新手支持度上显著优于GPT-3.5和Llama2（Mann-Whitney U统计量分别为300.5和322.5，p值分别为0.0017和0.0003）。该研究凸显了LLM为新手程序员生成定制化代码注释的潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show promise in generating code comments for novice programmers, but their educational effectiveness remains under-evaluated. This study assesses the instructional quality of code comments produced by GPT-4, GPT-3.5-Turbo, and Llama2, compared to expert-developed comments, focusing on their suitability for novices. Analyzing a dataset of ``easy'' level Java solutions from LeetCode, we find that GPT-4 exhibits comparable quality to expert comments in aspects critical for beginners, such as clarity, beginner-friendliness, concept elucidation, and step-by-step guidance. GPT-4 outperforms Llama2 in discussing complexity (chi-square = 11.40, p = 0.001) and is perceived as significantly more supportive for beginners than GPT-3.5 and Llama2 with Mann-Whitney U-statistics = 300.5 and 322.5, p = 0.0017 and 0.0003). This study highlights the potential of LLMs for generating code comments tailored to novice programmers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。