用大模型自动发现学生代码解释中的漏洞和误解。
Can LLMs Identify Gaps and Misconceptions in Students' Code Explanations?
- 对比零样本、微调与偏好优化三种方法识别代码解释缺陷。
- GPT-4在零样本下表现最优，微调模型优于提示工程。
- 偏好优化（ORPO）比微调更有效，适合教育评估场景。
本研究探索使用大型语言模型（LLMs）识别学生对代码示例自解释中存在的知识漏洞和概念误解。这是我们在自动化评估学生自由生成回答过程中的一部分，重点关注学生在理解代码活动中的自解释内容。我们实验了零样本提示、监督微调（SFT）以及偏好对齐方法来识别解释中的漏洞。人类评估证实，仅用简单提示，GPT-4在识别漏洞和误解方面持续优于LLaMA3和Mistral。此外，结果表明，经过微调的大型语言模型在识别学生解释中的漏洞方面，优于零样本和少样本提示方法。进一步发现，采用奇数比偏好优化（ORPO）的偏好对齐方法，在识别代码解释中的漏洞和误解方面优于监督微调。
原文摘要 · Abstract (English)
This paper investigates various approaches using Large Language Models (LLMs) to identify gaps and misconceptions in students' self-explanations of specific instructional material, in our case explanations of code examples. This research is a part of our larger effort to automate the assessment of students' freely generated responses, focusing specifically on their self-explanations of code examples during activities related to code comprehension. In this work, we experiment with zero-shot prompting, Supervised Fine-Tuning (SFT), and preference alignment of LLMs to identify gaps in students' self-explanation. With simple prompting, GPT-4 consistently outperformed LLaMA3 and Mistral in identifying gaps and misconceptions, as confirmed by human evaluations. Additionally, our results suggest that fine-tuned large language models are more effective at identifying gaps in students' explanations compared to zero-shot and few-shot prompting techniques. Furthermore, our findings show that the preference optimization approach using Odds Ratio Preference Optimization (ORPO) outperforms SFT in identifying gaps and misconceptions in students' code explanations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。