用AI生成的评语辅助助教批改经济学论文,提升效率与一致性。
Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use
- 基于助教评分标准,用大模型生成可交互的评语建议。
- 助教认为AI评语能提速、提准,但需详细评分细则支撑。
- 适合教育科技研究者与高校教学改革实践者参考。
本研究探讨将AI生成的反馈作为建议,以加速并提升人工教师批改质量的可能性。聚焦于一门基础大学经济学课程,该课程频繁布置短篇论文作业。我们开发了一个基于大模型的反馈引擎,根据助教使用的评分标准生成学生论文的反馈。为确保助教能有效评估和互动,他们完成了正常的批改任务。对随机抽取的已批改论文,我们利用反馈引擎生成评语,并以文内注释形式展示在Word文档中。通过20次、每次1小时的思考过程访谈,5位助教评估了AI反馈质量,对比其手写反馈,并分享若获建议将如何使用。研究强调,为知识密集型论文生成高质量反馈,必须提供详尽评分标准。助教认为,将AI反馈作为建议融入批改流程,可加快进度、提升一致性、改善整体反馈质量。我们还指出,应分步分解反馈生成任务并呈现中间结果,才能让助教有效利用AI反馈。
原文摘要 · Abstract (English)
This project examines the prospect of using AI-generated feedback as suggestions to expedite and enhance human instructors' feedback provision. In particular, we focus on understanding the teaching assistants' perspectives on the quality of AI-generated feedback and how they may or may not utilize AI feedback in their own workflows. We situate our work in a foundational college Economics class, which has frequent short essay assignments. We developed an LLM-powered feedback engine that generates feedback on students' essays based on grading rubrics used by the teaching assistants (TAs). To ensure that TAs can meaningfully critique and engage with the AI feedback, we had them complete their regular grading jobs. For a randomly selected set of essays that they had graded, we used our feedback engine to generate feedback and displayed the feedback as in-text comments in a Word document. We then performed think-aloud studies with 5 TAs over 20 1-hour sessions to have them evaluate the AI feedback, contrast the AI feedback with their handwritten feedback, and share how they envision using the AI feedback if they were offered as suggestions. The study highlights the importance of providing detailed rubrics for AI to generate high-quality feedback for knowledge-intensive essays. TAs considered that using AI feedback as suggestions during their grading could expedite grading, enhance consistency, and improve overall feedback quality. We discuss the importance of decomposing the feedback generation task into steps and presenting intermediate results, in order for TAs to use the AI feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。