用文本特征预测阅读题难度，准确率超基准30%。
Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository
- 构建含语言、测试和上下文特征的标注题库，用于难度预测。
- 模型预测误差（RMSE）降至0.59，相关性达0.77。
- 仅用语言特征或大模型嵌入即可实现良好效果，适合教育评估者使用。
基于文本内容预测阅读题难度具有重要意义。本文聚焦于从原始报告的题目正确率（p值）中恢复基于项目反应理论（IRT）的难度参数。我们构建了一个涵盖2018至2023年美国纽约与德克萨斯州3至8年级标准化考试的题库，包含阅读材料与学生数据，并标注了三类元数据：（1）题目语言特征，（2）文章测试特征，（3）上下文特征。采用带正则化的回归模型，融合全部特征后，预测难度的均方根误差（RMSE）为0.59，优于基线的0.92；真实难度与预测值的相关系数达0.77。进一步引入ModernBERT、BERT与LlAMA的文本嵌入，带来轻微性能提升。当仅使用语言特征或大模型嵌入时，预测表现相近，表明两类特征中任一即可满足需求。该模型可用于阅读题目的筛选与分类，将公开发布供各方使用。
原文摘要 · Abstract (English)
Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2018-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.59 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。