arXiv:2509.21910cs.CLcs.AI2025-09AAAI被引 18

用多智能体结构识别评分要素,提升AI作文批改准确性和可解释性。

AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition

  • 分两步:先提取评分要点,再据此打分,模仿人工阅卷流程。
  • 在4个数据集上,对齐率和相关性均优于单智能体模型,小模型提升更明显。
  • 适合教育评估、智能辅导系统,尤其适用于复杂评分标准场景。

自动化评分在教育中至关重要,可减少对人工评卷的依赖,实现学生作业的快速即时评价。尽管大语言模型(LLMs)在此任务中展现出巨大潜力,但作为端到端评分器仍面临准确性低、提示敏感、可解释性差和评分量表不匹配等问题,限制了其在实际评估中的应用。为此,我们提出AutoSCORE,一种基于评分量表对齐的结构化组件识别的多智能体大语言模型框架。该框架包含两个智能体:首先由评分要素提取智能体从学生作答中提取与评分量表相关的成分并编码为结构化表示;随后由评分智能体据此生成最终得分。该设计确保模型推理过程符合人类评分逻辑,提升了可解释性与鲁棒性。我们在ASAP基准的四个数据集上评估了AutoSCORE,使用GPT-4o、LLaMA-3.1-8B和LLaMA-3.1-70B等专有及开源模型。在多种任务与评分量表下,AutoSCORE始终优于单智能体基线,在评分一致性(QWK)、相关性、均方误差(MAE)和均方根误差(RMSE)等指标上均有提升,尤其在复杂多维评分量表上表现突出,且对小型模型有显著相对增益。结果表明,结合结构化组件识别与多智能体设计,可为自动化评分提供一种可扩展、可靠且可解释的解决方案。

原文摘要 · Abstract (English)

Automated scoring plays a crucial role in education by reducing the reliance on human raters, offering scalable and immediate evaluation of student work. While large language models (LLMs) have shown strong potential in this task, their use as end-to-end raters faces challenges such as low accuracy, prompt sensitivity, limited interpretability, and rubric misalignment. These issues hinder the implementation of LLM-based automated scoring in assessment practice. To address the limitations, we propose AutoSCORE, a multi-agent LLM framework enhancing automated scoring via rubric-aligned Structured COmponent REcognition. With two agents, AutoSCORE first extracts rubric-relevant components from student responses and encodes them into a structured representation (i.e., Scoring Rubric Component Extraction Agent), which is then used to assign final scores (i.e., Scoring Agent). This design ensures that model reasoning follows a human-like grading process, enhancing interpretability and robustness. We evaluate AutoSCORE on four benchmark datasets from the ASAP benchmark, using both proprietary and open-source LLMs (GPT-4o, LLaMA-3.1-8B, and LLaMA-3.1-70B). Across diverse tasks and rubrics, AutoSCORE consistently improves scoring accuracy, human-machine agreement (QWK, correlations), and error metrics (MAE, RMSE) compared to single-agent baselines, with particularly strong benefits on complex, multi-dimensional rubrics, and especially large relative gains on smaller LLMs. These results demonstrate that structured component recognition combined with multi-agent design offers a scalable, reliable, and interpretable solution for automated scoring.

自动化评分多智能体大模型教育AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。