为中小学科学教材自动评估建基准,验证大模型需领域微调才有效。
SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials

- 构建首个教材评估基准SciEval,含273节课程、13项指标共3549条评分
- 主流大模型在该任务上表现不佳,微调后性能提升最高达11%
- 适合教育AI研发者、教材评测人员及生成式教学工具开发者
随着生成式AI在中小学科学教学材料中的广泛应用,对教学材料的评估需求日益迫切。但传统人工评估耗时费力且难规模化,推动自动化评估的发展。尽管大语言模型在通用任务中表现优异,其在教学材料评估上的可靠性尚不明确。为此,本文将自动教学材料评估(AIME)定义为生成式任务,基于教师设计的评价量表预测评分与证据。构建首个AIME基准数据集SciEval,包含273节课程级教学材料,由专家依据EQuIP量表在13个维度上标注,共3549条评分,具有高一致性。测试GPT、Gemini、Llama和Qwen等主流模型发现均表现有限。对Qwen3进行领域微调后,在保留测试集上性能提升最高达11%,表明领域适配微调对AIME至关重要,也为大模型在其他教育任务中的应用提供支持。
原文摘要 · Abstract (English)
The need to evaluate instructional materials for K-12 science education has become increasingly important, as more educators use generative AI to create instructional materials. However, the review of instructional materials is time-consuming, expertise-intensive, and difficult to scale, motivating interest in automated evaluation approaches. While large language models (LLMs) have shown strong performance on general evaluation tasks, their performance and reliability on instructional materials remain unclear. To address this gap, we formulate Automatic Instructional Materials Evaluation (AIME) as a generative AI task that predicts scores and evidence using the rubric designed by the educator. We create a benchmark dataset and develop baseline models for AIME. First, we curate the first AIME dataset, SciEval, consisting of instructional materials annotated with pedagogy-aligned evaluation scores and evidence-based rationales. Expert annotations achieve high inter-rater reliability, resulting in a dataset of 273 lesson-level instructional materials evaluated across 13 criteria (N=3549) using the EQuIP rubric. Second, we test mainstream LLMs (GPT, Gemini, Llama, and Qwen) on SciEval and find that none achieve strong performance. Then we fine-tune Qwen3 on SciEval. Results on a held-out test set show that domain-aligned fine-tuning can achieve up to 11 percent performance gains, highlighting the importance of domain-specific fine-tuning for AIME and facilitating the use of LLMs in other educational tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。