用大模型生成评语，自动预测试题能否通过审核
Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques
- 结合原始题目文本与大模型生成的评语进行融合判断
- 整体准确率达75%，数学题预测效果优于语文题
- 适合自动化出题场景，但公平性问题仍需人工把关
自动化项目评估（AIE）指不依赖人工专家评审或实际测试，仅通过计算方法评估题目质量。本研究基于大规模标准化考试的历史拒用数据，构建了一个近全面的AIE模型，旨在从题目文本预测其是否被接受或拒绝。数据集包含52,759道英语与数学题目，其中34%被永久拒用，拒用原因包括测量学属性差、内容问题、偏见与敏感性问题及非内容问题。我们对DeBERTaV3-large分类器在原始题目文本上进行微调，另对Qwen3生成的题目评语使用第二个DeBERTa分类器，并构建融合模型。融合模型表现最优（准确率=0.75，F1=0.64，AUC=0.80，灵敏度=0.64，特异性=0.81）。数学题预测更准（F1=0.73，AUC=0.86），语文题较差（F1=0.51，AUC=0.72）。将决策阈值从0.5降至0.25可使语文和数学的平均灵敏度分别提升至0.88和0.91，但特异性下降至0.31和0.56，更适合低成本自动生成题目场景。结合评语能提升多数拒用类型的表现。模型对难度更高的题目赋予更高拒用概率。但融合模型在识别偏见、敏感性、公平性和可访问性问题上表现不佳，尤其在语文题中。结果表明，基于文本的AIE在某些领域可行，可减轻人工评审负担，但仍强调对公平性相关题目需保留人工审查。
原文摘要 · Abstract (English)
Automated item evaluation (AIE) refers to the use of computational methods to assess item quality without requiring manual expert review or field testing of the items under evaluation. We aimed to build a near-comprehensive AIE model by predicting item acceptance and rejection from item text using historical rejection data from a large-scale standardized testing program. The dataset contained 52,759 English language arts (ELA) and mathematics items with 34% permanently rejected from future operational use. Rejection reasons included poor psychometric properties, content issues, bias and sensitivity concerns, and non-content issues. We fine-tuned a DeBERTaV3-large classifier on raw item text, a second DeBERTa classifier on Qwen3-generated item critiques, and a fusion model combining representations from both. The fusion model achieved the strongest overall performance (Accuracy = .75, F1 = .64, AUC = .80, Sensitivity = .64, Specificity = .81). Prediction for math (F1 = .73, AUC = .86) was considerably more accurate than ELA (F1 = .51, AUC = .72). Lowering the decision threshold from .5 to .25 raised average sensitivity for ELA and math to .88 and .91, while reducing specificity to .31 and .56, respectively, which may be preferable in automated item generation contexts where generating items is cheaper than evaluating them. Incorporating item critiques alongside raw item text improved performance across most rejection reasons. The model assigned higher rejection probabilities to more difficult items. However, the fusion model struggled to identify items flagged for bias, sensitivity, fairness, or accessibility, especially for ELA. These findings suggest that text-based AIE is feasible in some areas and may offer a practical tool for reducing the burden of manual review and field testing, while also underscoring the importance of human review for items with fairness concerns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。