用大模型比作文好坏，自动打分更准更省力
LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models
- 让大模型比较两篇作文优劣，再转成分数
- 在多个测试集上准确率超越传统零样本方法
- 适合作文自动评分、教育评测等场景
大语言模型（LLMs）的进展使得零样本自动作文评分（AES）成为可能，显著降低人工评分的成本与工作量。然而，现有零样本方法多依赖模型直接生成绝对分数，常因模型偏见和评分不一致而偏离人类评价。为此，我们提出基于大模型的作文对比评分方法（LCES），将AES建模为成对比较任务：指导LLM判断两篇作文中哪篇更优，收集大量比较结果，并通过RankNet高效转换为连续分数。由于比较数量随作文数平方增长，我们采用RankNet提升可扩展性。在多个标准AES数据集上的实验表明，LCES在保持计算效率的同时，准确率优于传统零样本方法。此外，该方法对不同大模型底座具有鲁棒性，具备实际应用潜力。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have enabled zero-shot automated essay scoring (AES), providing a promising way to reduce the cost and effort of essay scoring in comparison with manual grading. However, most existing zero-shot approaches rely on LLMs to directly generate absolute scores, which often diverge from human evaluations owing to model biases and inconsistent scoring. To address these limitations, we propose LLM-based Comparative Essay Scoring (LCES), a method that formulates AES as a pairwise comparison task. Specifically, we instruct LLMs to judge which of two essays is better, collect many such comparisons, and convert them into continuous scores. Considering that the number of possible comparisons grows quadratically with the number of essays, we improve scalability by employing RankNet to efficiently transform LLM preferences into scalar scores. Experiments using AES benchmark datasets show that LCES outperforms conventional zero-shot methods in accuracy while maintaining computational efficiency. Moreover, LCES is robust across different LLM backbones, highlighting its applicability to real-world zero-shot AES.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。