首个巴斯克语作文评分与反馈数据集,助力低资源语言AI教育研究
Automatic Essay Scoring and Feedback Generation in Basque Language Learning
- 构建3200篇巴斯克语作文数据集,涵盖5项评分维度及详细反馈
- 微调Latxa模型后,评分一致性和反馈质量超越GPT-5等闭源系统
- 提出新评估方法,结合自动指标与专家验证,确保反馈教学价值
本文首次发布面向巴斯克语的自动作文评分(AES)与反馈生成公开数据集,针对欧洲共同语言参考框架(CEFR)C1水平。数据集包含3,200篇来自HABE的作文,由专家标注正确性、丰富性、连贯性、衔接性与任务契合度五项指标,并附详细反馈与错误示例。我们对RoBERTa-EusCrawl和Latxa 8B/70B等开源模型进行微调,用于评分与解释生成。实验表明,编码器模型在评分上依然可靠,而经过监督微调(SFT)的Latxa显著提升性能,在评分一致性与反馈质量上超越GPT-5、Claude Sonnet 4.5等闭源先进系统。我们还提出一种新型反馈生成评估方法,结合自动一致性指标与专家对提取错误的验证。结果表明,微调后的Latxa能生成与评分标准对齐、具教学意义的反馈,并识别更广泛的错误类型。该资源与基准为巴斯克语等低资源语言的透明、可复现、教育导向的NLP研究奠定基础。
原文摘要 · Abstract (English)
This paper introduces the first publicly available dataset for Automatic Essay Scoring (AES) and feedback generation in Basque, targeting the CEFR C1 proficiency level. The dataset comprises 3,200 essays from HABE, each annotated by expert evaluators with criterion specific scores covering correctness, richness, coherence, cohesion, and task alignment enriched with detailed feedback and error examples. We fine-tune open-source models, including RoBERTa-EusCrawl and Latxa 8B/70B, for both scoring and explanation generation. Our experiments show that encoder models remain highly reliable for AES, while supervised fine-tuning (SFT) of Latxa significantly enhances performance, surpassing state-of-the-art (SoTA) closed-source systems such as GPT-5 and Claude Sonnet 4.5 in scoring consistency and feedback quality. We also propose a novel evaluation methodology for assessing feedback generation, combining automatic consistency metrics with expert-based validation of extracted learner errors. Results demonstrate that the fine-tuned Latxa model produces criterion-aligned, pedagogically meaningful feedback and identifies a wider range of error types than proprietary models. This resource and benchmark establish a foundation for transparent, reproducible, and educationally grounded NLP research in low-resource languages such as Basque.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。