用专家与学生代理协作生成高质量编程题,减少人工干预。
Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents
- 分阶段用强弱模型模拟专家与学生,自动验证题目质量
- 生成任务与专家设计任务在质量上相当,且更易懂更吸引人
- 适合教育平台自动化出题,降低教师负担
生成式AI正在改变计算教育,实现个性化内容与反馈的自动生成。我们研究其在为学生生成高质量编程任务方面的能力。尽管任务生成进展显著,但AI生成任务与专家设计任务之间仍存在质量差距:可能偏离目标编程概念、对学生不清晰,或包含错误测试。现有方法常需教师手动验证。为此,我们提出PyTaskSyn,一种新型合成技术:先生成编程任务,再通过多阶段评估判断是否达标。核心思路是利用强弱生成模型模拟专家与学生代理,协同完成任务生成与验证。大量评估表明,PyTaskSyn显著提升任务质量,且各代理角色不可或缺。此外,通过公开的网页应用进行用户研究,结果表明该方法生成的任务质量接近专家设计,同时降低工作量和成本,并比在线资源中的任务更具吸引力。
原文摘要 · Abstract (English)
Generative AI is transforming computing education by enabling the automatic generation of personalized content and feedback. We investigate its capabilities in providing high-quality programming tasks to students. Despite promising advancements in task generation, a quality gap remains between AI-generated and expert-created tasks. The AI-generated tasks may not align with target programming concepts, could be incomprehensible to students, or may contain critical issues such as incorrect tests. Existing works often require interventions from human teachers for validation. We address these challenges by introducing PyTaskSyn, a novel synthesis technique that first generates a programming task and then decides whether it meets certain quality criteria to be given to students. The key idea is to break this process into multiple stages performed by expert and student agents simulated using both strong and weaker generative models. Through extensive evaluation, we show that PyTaskSyn significantly improves task quality compared to baseline techniques and showcases the importance of each specialized agent type in our validation pipeline. Additionally, we conducted user studies using our publicly available web application and show that PyTaskSyn can deliver high-quality programming tasks comparable to expert-designed ones while reducing workload and costs, and being more engaging than programming tasks that are available in online resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。