用大模型分析真实辅导对话，自动识别并评估教师行为有效性。
Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study
- 用多个大模型分析50段远程辅导对话，识别表扬与纠错行为。
- 对表扬和错误回应的判断准确率达82%-98%，接近人工水平。
- 提出低成本提示策略，适合教育研究与智能辅导系统开发。
辅导能提升学生成绩，但基于语音转录文本大规模识别与研究哪些辅导行为最有助于学习仍是开放问题。本研究探索使用生成式AI识别并评估真实数学辅导中教师行为的可行性与可扩展性。我们分析了50段随机选取的大学生成人远程辅导初中生数学的对话转录。利用GPT-4、GPT-4o、GPT-4-turbo、Gemini-1.5-pro和LearnLM，评估教师在两个关键技能上的表现：给予有效表扬和应对学生数学错误。所有模型均可靠检测相关情境，如教师给予表扬（准确率94%-98%）和学生出现数学错误（准确率82%-88%），并有效评估其是否符合教学最佳实践，与人工判断高度一致（对应准确率分别为83%-89%和73%-77%）。我们提出一种成本效益高的提示策略，并讨论其在真实场景中支持规模化评估的实际意义。本研究还提供了可复现的LLM提示模板，推动人工智能支持学习的研究发展。
原文摘要 · Abstract (English)
Tutoring improves student achievement, but identifying and studying what tutoring actions are most associated with student learning at scale based on audio transcriptions is an open research problem. This present study investigates the feasibility and scalability of using generative AI to identify and evaluate specific tutor moves in real-life math tutoring. We analyze 50 randomly selected transcripts of college-student remote tutors assisting middle school students in mathematics. Using GPT-4, GPT-4o, GPT-4-turbo, Gemini-1.5-pro, and LearnLM, we assess tutors' application of two tutor skills: delivering effective praise and responding to student math errors. All models reliably detected relevant situations, for example, tutors providing praise to students (94-98% accuracy) and a student making a math error (82-88% accuracy) and effectively evaluated the tutors' adherence to tutoring best practices, aligning closely with human judgments (83-89% and 73-77%, respectively). We propose a cost-effective prompting strategy and discuss practical implications for using large language models to support scalable assessment in authentic settings. This work further contributes LLM prompts to support reproducibility and research in AI-supported learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。