用检索增强生成让大模型精准批改简答题，还能给出分点反馈。
SteLLA: A Structured Grading System Using LLMs with RAG
- 基于参考答案和评分标准，用RAG从外部知识中提取结构化信息增强模型
- 大模型按问题逐项评估学生作答，实现与人工评分高度一致的分项打分
- 适合需要细粒度反馈的教育场景，尤其适合自动化作业/考试批改
大型语言模型在诸多任务中展现出强大的通用能力，但在自动化简答题评分（ASAG）等具体任务中仍面临可靠性挑战。本文提出SteLLA（基于RAG的结构化评分系统），通过检索增强生成技术，从与教师提供的参考答案和评分标准高度相关的外部知识中提取结构化信息，增强大模型在该任务上的表现；同时，利用大模型对学生的答案进行结构化、问答式评估，实现对所有考查知识点的分项评分与反馈。实验基于某大学生物学课程的真实考试数据集开展，结果表明系统与人工评分具有显著一致性，并能提供全面的分项成绩与反馈。对GPT4生成反馈的定性分析显示，其善于捕捉事实，但可能过度推断文本隐含含义，为大模型在自动评分中的应用提供了重要启示。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown strong general capabilities in many applications. However, how to make them reliable tools for some specific tasks such as automated short answer grading (ASAG) remains a challenge. We present SteLLA (Structured Grading System Using LLMs with RAG) in which a) Retrieval Augmented Generation (RAG) approach is used to empower LLMs specifically on the ASAG task by extracting structured information from the highly relevant and reliable external knowledge based on the instructor-provided reference answer and rubric, b) an LLM performs a structured and question-answering-based evaluation of student answers to provide analytical grades and feedback. A real-world dataset that contains students' answers in an exam was collected from a college-level Biology course. Experiments show that our proposed system can achieve substantial agreement with the human grader while providing break-down grades and feedback on all the knowledge points examined in the problem. A qualitative and error analysis of the feedback generated by GPT4 shows that GPT4 is good at capturing facts while may be prone to inferring too much implication from the given text in the grading task which provides insights into the usage of LLMs in the ASAG system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。