arXiv:2606.14113cs.SEcs.CL2026-06

用大模型模拟学生编程错误,高效生成真实可测的代码错误数据。

Simulating Students' Java Programming Errors with Large Language Models

论文配图:Simulating Students' Java Programming Errors with Large Language Models
图 1 · 摘自论文原文
  • 用大模型在37个编程题上生成模拟错误,测试三种提示策略效果。
  • 最佳模型生成错误种类多且与真实学生错误高度一致,相似度达92%。
  • 适合教育研究者、智能辅导系统开发者使用,尤其关注学习难点分析。

理解学生编程错误是编程教育的核心,但获取新任务下的代表性错误样本耗时费力,因真实提交需长期教学积累。本文探索大语言模型(LLMs)能否作为可扩展的学生代理,生成真实的逻辑错误。基于包含74,000+唯一学生Java提交的CodeWorkout数据集,评估5个主流大模型在输入输出(IO)、思维链(CoT)和迭代自修正(Self-Refine)三种提示策略下的表现。从错误模式多样性与真实学生错误对齐度两个维度进行评估,并分析不同难度任务的影响。结果表明:所有模型均能生成多样错误,但对齐度差异显著,Claude Sonnet 4表现最均衡。通过盲评专家标注(N=401),合成错误与真实错误在功能上无法区分。高难度任务产生更多样但偏离真实学生的错误。研究揭示了使用大模型模拟学习者的权衡,为教学智能体、智能辅导系统和大规模学习分析提供了设计启示。

原文摘要 · Abstract (English)

Understanding student errors in the programming is a cornerstone of programming education, yet obtaining a representative set of student errors for any newly designed task remains slow and costly, since authentic submissions only accumulate after extensive classroom deployment. This paper explores whether large language models (LLMs) can serve as scalable proxies for students by simulating realistic logical errors in code submissions. Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, we evaluate five LLMs under three mainstream prompting strategies: Input-Output (IO), Chain-of-Thought (CoT), and iterative Self-Refine. We assess performance along two key dimensions: diversity (the range of distinct error patterns) and alignment (alignment with authentic student mistakes), and examine how these vary by struggling level of programming tasks. Our quantitative findings reveal that while all models generate diverse errors, their alignment to human submissions diverges: Claude Sonnet 4 achieves the most balanced performance. In addition, we conducted a blinded expert annotation study (N = 401) comparing synthetic and authentic errors. This qualitative analysis confirms that the generated errors are functionally indistinguishable from authentic student errors. Moreover, higher-struggling-level problems elicit more diverse but less student-like errors. These results highlight trade-offs in using LLMs to simulate human learners and suggest design considerations for integrating synthetic errors into teachable agents, intelligent tutoring systems, and large-scale learning analytics.

编程教育大模型错误模拟智能辅导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。