用多模型嵌入融合+梯度提升，大幅提高长作文自动评分准确率
Empirical Comparison of Encoder-Based Language Models and Feature-Based Supervised Machine Learning Approaches to Automated Scoring of Long Essays
- 融合多个预训练模型的文本嵌入特征，再用梯度提升分类器集成
- 集成模型在1.7万篇作文上达0.829的加权肯德尔系数，显著优于单模型
- 适合需要高精度作文评分的教育评估系统开发者
长文本处理对仅编码器的语言模型构成挑战，尤其在作文自动评分任务中。本研究训练了多种常用编码器模型（BERT、RoBERTa、DistilBERT、DeBERTa）用于长作文评分，并与基于512个标记限制的基模型构建的集成模型进行对比。实验包含多模型嵌入集成、以及基于梯度提升决策树、XGBoost和LightGBM等特征驱动的监督学习模型集成。所有模型在17,307篇作文的数据集上按80%/10%/10%划分进行训练、验证和测试，使用加权肯德尔一致性系数（Quadratic Weighted Kappa）评估性能。结果表明，结合多个预训练语言模型表示并以梯度提升分类器作为集成器的嵌入集成模型，在长作文评分任务中显著优于单一语言模型。
原文摘要 · Abstract (English)
Long context may impose challenges for encoder-only language models in text processing, specifically for automated scoring of essays. This study trained several commonly used encoder-based language models for automated scoring of long essays. The performance of these trained models was evaluated and compared with the ensemble models built upon the base language models with a token limit of 512?. The experimented models include BERT-based models (BERT, RoBERTa, DistilBERT, and DeBERTa), ensemble models integrating embeddings from multiple encoder models, and ensemble models of feature-based supervised machine learning models, including Gradient-Boosted Decision Trees, eXtreme Gradient Boosting, and Light Gradient Boosting Machine. We trained, validated, and tested each model on a dataset of 17,307 essays, with an 80%/10%/10% split, and evaluated model performance using Quadratic Weighted Kappa. This study revealed that an ensemble-of-embeddings model that combines multiple pre-trained language model representations with gradient-boosting classifier as the ensemble model significantly outperforms individual language models at scoring long essays.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。