测试五款大模型在高校作文评分中的可靠性，发现其表现远不如人类。
Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education
- 用五款大模型对67篇心理学论文评分，每篇评三次看稳定性
- 模型间与人类评分差异大，相关性不显著，整体一致性低于0.3
- 模型易高估连贯性，对专业判断要求高的维度（如可行性）表现差
本研究评估了五款先进大语言模型（Claude 3.5、DeepSeek v2、Gemini 2.5、GPT-4 和 Mistral 24B）在真实高等教育场景下自动作文评分的可靠性和有效性。共分析了67篇意大利语学生作文，这些作文是某大学心理学课程的作业，采用四项评分标准（相关性、连贯性、原创性、可行性）。每种模型对所有文章进行三次提示复现以评估模型内稳定性。结果显示，人机评分一致性低且不显著（加权平方卡帕），模型内部重复评分的一致性同样较弱（中位数肯德尔W < 0.30）。系统性偏差显现，如普遍高估连贯性，对依赖上下文的维度处理不一致。跨模型一致性分析显示，连贯性和原创性有中等程度收敛，但在相关性和可行性上几乎无共识。尽管样本有限，结果表明当前大模型在需要学科洞察和情境敏感的任务中难以复制人类判断。在评估开放式学术写作时，尤其在解释性领域，人工监督仍至关重要。
原文摘要 · Abstract (English)
This study investigates the reliability and validity of five advanced Large Language Models (LLMs), Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral 24B, for automated essay scoring in a real world higher education context. A total of 67 Italian-language student essays, written as part of a university psychology course, were evaluated using a four-criterion rubric (Pertinence, Coherence, Originality, Feasibility). Each model scored all essays across three prompt replications to assess intra-model stability. Human-LLM agreement was consistently low and non-significant (Quadratic Weighted Kappa), and within-model reliability across replications was similarly weak (median Kendall's W < 0.30). Systematic scoring divergences emerged, including a tendency to inflate Coherence and inconsistent handling of context-dependent dimensions. Inter-model agreement analysis revealed moderate convergence for Coherence and Originality, but negligible concordance for Pertinence and Feasibility. Although limited in scope, these findings suggest that current LLMs may struggle to replicate human judgment in tasks requiring disciplinary insight and contextual sensitivity. Human oversight remains critical when evaluating open-ended academic work, particularly in interpretive domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。