将人口统计信息直接拼接进文本模型会降低作文评分准确率。
Demographic Metadata as Construct-Irrelevant Noise in DistilBERT-Based Automated Essay Scoring
- 用简单拼接方式融合人口信息与文本输入
- 模型准确率从0.727降至0.656，验证损失上升
- 加剧评分偏差，公平性测试通过数减少
自动作文评分（AES）系统正被广泛用于减轻教师负担和支撑大规模评估。尽管人类评分常受学生人口特征影响，但将这些元数据与文本输入结合的策略效果仍不明确。本研究考察了在基于DistilBERT的模型中，采用朴素的多模态融合策略——直接拼接已分词的文本与人口统计信息——对预测精度、训练收敛性和评分公平性的影响。使用ASAP 2.0数据集进行10折交叉验证，对比基线模型与加入元数据的实验模型。结果表明，早期融合人口信息显著降低整体预测精度：基线模型的加权二次κ值（QWK）为0.727，引入元数据后降至0.656；实验模型验证损失升至1.29（基线为1.25）。此外，评分偏差加剧，公平性测试通过数从15减至12/19次。
原文摘要 · Abstract (English)
Automated Essay Scoring (AES) systems are increasingly used to support teachers in managing grading workloads and to provide a supplementary rater in large-scale assessments. While human grading is frequently influenced by students' demographic characteristics, the efficacy of different strategies for integrating demographic metadata with textual input used to train AES models remains underexplored. This study investigates the impact of a specific multimodal fusion strategy - naive metadata concatenation - on the predictive accuracy, training convergence, and score parity of a DistilBERT-based AES model. A comparative analysis was conducted using the ASAP 2.0 dataset to evaluate a baseline model against an experimental model trained with input that concatenates tokenised text and demographic metadata using a naive multimodal fusion strategy. Evaluated via 10-fold cross-validation, the findings reveal that the early fusion of demographic metadata and the input significantly degrades the model's overall predictive accuracy. The baseline model achieved a Quadratic Weighted Kappa (QWK) of 0.727, which dropped to 0.656 upon integrating metadata. Furthermore, the experimental model exhibited higher validation loss (1.29) compared to the baseline model (1.25). The experimental model also displayed exacerbated scoring bias, reducing score parity instances from 15 to 12 out of 19 tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。