用AI自动评分学术文本题,参考答案加持效果最好。
Analysis of instruction-based LLMs' capabilities to score and judge text-input problems in an academic setting
- 引入参考答案引导评分,提升AI评估准确性。
- 结合人工评分,参考答案法误差最小(中位绝对偏差0.945)。
- 适合教育领域批量作业批改,辅助教师工作。
大型语言模型(LLMs)可作为评价工具,在教育领域被用于辅助师生。本研究针对高等教育计算机科学类文本问答题,构建了五个基于LLM的自动评分系统,并在包含110个学生作答的自定义数据集上测试了JudgeLM、Llama-3.1-8B和DeepSeek-R1-Distill-Llama-8B三个模型。评价方法包括:仅使用模型单次输出的JudgeLM评估、引入正确答案作为参考的参考答案法、不使用参考答案的无参考评估、基于原子标准的加法评估,以及根据问题动态生成标准的自适应评估。所有方法均与人工评分结果对比。结果显示,参考答案法表现最优,中位绝对偏差为0.945,均方根偏差为1.214,评分公正且全面。其他方法如加法与自适应评估在简洁回答中表现不佳,无参考评估因缺乏必要信息导致评分不准,JudgeLM因模型自身限制效果有限。结论表明,结合合理方法的AI自动评分系统具备作为教学辅助工具的潜力。
原文摘要 · Abstract (English)
Large language models (LLMs) can act as evaluators, a role studied by methods like LLM-as-a-Judge and fine-tuned judging LLMs. In the field of education, LLMs have been studied as assistant tools for students and teachers. Our research investigates LLM-driven automatic evaluation systems for academic Text-Input Problems using rubrics. We propose five evaluation systems that have been tested on a custom dataset of 110 answers about computer science from higher education students with three models: JudgeLM, Llama-3.1-8B and DeepSeek-R1-Distill-Llama-8B. The evaluation systems include: The JudgeLM evaluation, which uses the model's single answer prompt to obtain a score; Reference Aided Evaluation, which uses a correct answer as a guide aside from the original context of the question; No Reference Evaluation, which ommits the reference answer; Additive Evaluation, which uses atomic criteria; and Adaptive Evaluation, which is an evaluation done with generated criteria fitted to each question. All evaluation methods have been compared with the results of a human evaluator. Results show that the best method to automatically evaluate and score Text-Input Problems using LLMs is Reference Aided Evaluation. With the lowest median absolute deviation (0.945) and the lowest root mean square deviation (1.214) when compared to human evaluation, Reference Aided Evaluation offers fair scoring as well as insightful and complete evaluations. Other methods such as Additive and Adaptive Evaluation fail to provide good results in concise answers, No Reference Evaluation lacks information needed to correctly assess questions and JudgeLM Evaluations have not provided good results due to the model's limitations. As a result, we conclude that Artificial Intelligence-driven automatic evaluation systems, aided with proper methodologies, show potential to work as complementary tools to other academic resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。