新手写代码时失败，不是因为词汇少，而是不懂需要提供多少信息。
Substance Beats Style: Why Beginning Students Fail to Code with LLMs
- 用实验证明：技术词汇不足只是表面现象
- 提示词的信息量才是决定成败的关键
- 适合编程教育研究者和AI辅助教学设计者
尽管大语言模型（LLMs）提升了专业程序员的效率，但现有研究显示，新手在使用LLM完成文本转代码任务时仍面临困难。本文探讨了两种可能的原因：一是学生缺乏必要的技术词汇来撰写有效提示，二是学生不了解生成代码所需的完整信息量。通过因果干预实验分析技术词汇的影响，并结合提示编辑过程的图谱分析不同错误类型。结果表明：技术词汇不足仅与提示失败相关，而提示的信息含量才是预测成功的决定因素；学生常陷入琐碎修改的困境。这些发现对编程教育中使用LLMs及推动计算普及具有重要启示。
原文摘要 · Abstract (English)
Although LLMs are increasing the productivity of professional programmers, existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks. Why is this the case? This paper explores two competing hypotheses about the cause of student-LLM miscommunication: (1) students simply lack the technical vocabulary needed to write good prompts, and (2) students do not understand the extent of information that LLMs need to solve code generation tasks. We study (1) with a causal intervention experiment on technical vocabulary and (2) by analyzing graphs that abstract how students edit prompts and the different failures that they encounter. We find that substance beats style: a poor grasp of technical vocabulary is merely correlated with prompt failure; that the information content of prompts predicts success; that students get stuck making trivial edits; and more. Our findings have implications for the use of LLMs in programming education, and for efforts to make computing more accessible with LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。