用大模型生成学生代码错误，经引导可逼近真实学生出错模式。
Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students
- 给大模型提供常见错误类型和频率描述，引导其生成逼真错误。
- 无引导时大模型生成的错误分布与真实学生差异大，多数不现实。
- 适合教育数据合成、冷启动教学工具开发的研究者使用。
大型语言模型（LLMs）为生成合成课堂数据提供了新机遇，可用于模拟包含典型错误分布的代码、解决教育工具开发中的冷启动问题，或在隐私受限时替代真实用户数据。本文通过对比分析大模型与计算机专业学生生成的错误分布，基于两项先前的大规模学生错误数据研究，探究能否通过提示使大模型生成类似真实学生的错误。结果表明，未经引导时，大模型生成的错误分布不具合理性，许多错误极不可能由真实学生写出；但通过提供常见错误类型及典型频率描述，大模型可被引导生成符合真实学生错误分布的合成代码。
原文摘要 · Abstract (English)
Large language models (LLMs) present an exciting opportunity for generating synthetic classroom data. Such data could include code containing a typical distribution of errors, simulated student behaviour to address the cold start problem when developing education tools, and synthetic user data when access to authentic data is restricted due to privacy reasons. In this research paper, we conduct a comparative study examining the distribution of bugs generated by LLMs in contrast to those produced by computing students. Leveraging data from two previous large-scale analyses of student-generated bugs, we investigate whether LLMs can be coaxed to exhibit bug patterns that are similar to authentic student bugs when prompted to inject errors into code. The results suggest that unguided, LLMs do not generate plausible error distributions, and many of the generated errors are unlikely to be generated by real students. However, with guidance including descriptions of common errors and typical frequencies, LLMs can be shepherded to generate realistic distributions of errors in synthetic code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。