研究教师如何修改AI生成的作业反馈,发现多数不改,改了也更简洁。
Understanding Teacher Revisions of Large Language Model-Generated Feedback
- 分析1349条反馈,发现教师80%直接采纳,改写后文本变长再缩短。
- 仅靠AI反馈文本就能预测是否被修改,准确率AUC达0.75。
- 改写多为简化内容,从详细解释转为简短纠错,契合教师实际需求。
大型语言模型(LLMs)越来越多地生成学生学业反馈,但教师如何在反馈传递给学生前进行修改仍不明确。教师的修订行为直接影响学生最终接收的内容,因此修订实践对评估AI教学工具至关重要。本研究分析了来自117名教师的1,349个AI生成反馈及其对应的教师修订版本。研究考察了:(i) 与教师修订相关的文本特征;(ii) 是否可仅通过AI反馈文本预测修订决策;(iii) 修订如何改变反馈的教育类型。首先,约80%的AI反馈未被修改,而被修改的反馈通常初始更长,随后被教师缩短。教师间差异显著:约50%从不修改,仅约10%修改超过三分之二的反馈实例。其次,仅使用句子嵌入作为输入特征的机器学习模型,在识别将被修改的反馈上达到中等表现(AUC=0.75)。第三,定性编码显示,当发生修订时,教师常简化内容,使反馈从高信息量解释转向更简洁、纠正性的形式。这些发现刻画了教师在实践中与AI生成反馈的互动方式,并指出了可设计更契合教师优先事项、减少冗余编辑的反馈系统。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly generate formative feedback for students, yet little is known about how teachers revise this feedback before it reaches learners. Teachers' revisions shape what students receive, making revision practices central to evaluating AI classroom tools. We analyze a dataset of 1,349 instances of AI-generated feedback and corresponding teacher-edited explanations from 117 teachers. We examine (i) textual characteristics associated with teacher revisions, (ii) whether revision decisions can be predicted from the AI feedback text, and (iii) how revisions change the pedagogical type of feedback delivered. First, we find that teachers accept AI feedback without modification in about 80% of cases, while edited feedback tends to be significantly longer and subsequently shortened by teachers. Editing behavior varies substantially across teachers: about 50% never edit AI feedback, and only about 10% edit more than two-thirds of feedback instances. Second, machine learning models trained only on the AI feedback text as input features, using sentence embeddings, achieve fair performance in identifying which feedback will be revised (AUC=0.75). Third, qualitative coding shows that when revisions occur, teachers often simplify AI-generated feedback, shifting it away from high-information explanations toward more concise, corrective forms. Together, these findings characterize how teachers engage with AI-generated feedback in practice and highlight opportunities to design feedback systems that better align with teacher priorities while reducing unnecessary editing effort.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。