LLM生成的错误提示对新手程序员帮助有限，实测效果不如人工编写。
Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
- 让106名新手学生修复6个真实代码错误，对比三种提示方式
- GPT-4生成提示仅在1个任务中比原生编译器提示更快修复
- 人工编写的错误解释仍最优，适合教学场景使用
大型语言模型（如ChatGPT）在编程教育领域引发变革。尽管已有研究显示其在解释和解决编译错误方面表现优异，但这些评估多基于专家在模拟环境中的测试。本研究通过一个包含106名参与者的组内实验，考察新手程序员在真实情境下如何处理编程错误消息（PEMs）。学生需修复6个含错的C程序，每题随机分配使用原始编译错误信息、人工编写的错误解释或GPT-4生成的解释。结果显示，尽管在合成基准上表现亮眼，但GPT-4生成的解释仅在1个任务中显著缩短了修复时间。人工编写的解释在客观和主观评价上均优于其他两种方式。
原文摘要 · Abstract (English)
The sudden emergence of large language models (LLMs) such as ChatGPT has had a disruptive impact throughout the computing education community. LLMs have been shown to excel at producing correct code to CS1 and CS2 problems, and can even act as friendly assistants to students learning how to code. Recent work shows that LLMs demonstrate unequivocally superior results in being able to explain and resolve compiler error messages -- for decades, one of the most frustrating parts of learning how to code. However, LLM-generated error message explanations have only been assessed by expert programmers in artificial conditions. This work sought to understand how novice programmers resolve programming error messages (PEMs) in a more realistic scenario. We ran a within-subjects study with $n$ = 106 participants in which students were tasked to fix six buggy C programs. For each program, participants were randomly assigned to fix the problem using either a stock compiler error message, an expert-handwritten error message, or an error message explanation generated by GPT-4. Despite promising evidence on synthetic benchmarks, we found that GPT-4 generated error messages outperformed conventional compiler error messages in only 1 of the 6 tasks, measured by students' time-to-fix each problem. Handwritten explanations still outperform LLM and conventional error messages, both on objective and subjective measures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。