用AutoML自动化校准语言测试题目的难度参数，提升评分准确性和模型表现。
AutoIRT: Calibrating Item Response Theory Models with Automated Machine Learning
- 分两阶段：先用AutoML预测题目得分，再拟合参数化模型
- 在杜尔林英语测试中表现优于传统IRT和BERT-IRT方法
- 无需复杂架构，适合快速部署于高阶自适应测试场景
项目反应理论（IRT）是一类可解释的因子模型，广泛用于计算机自适应测试（CAT），如语言能力测试。传统方法基于概率模型拟合考生答对题目的可能性，而神经网络扩展模型（如BertIRT）需定制架构与调参。本文提出一种多阶段拟合流程，兼容现成的自动化机器学习（AutoML）工具。该方法采用蒙特卡洛期望最大化（MCEM）外循环，结合双阶段内循环：第一阶段使用题目特征训练非参数化AutoML评分模型；第二阶段拟合针对每道题的参数化模型。该方法显著加速测试评分建模流程。我们在杜尔林英语测试（Duolingo English Test）上验证了其有效性，结果表明该模型通常更具校准性、预测性能更优、分数更准确，优于非解释性IRT模型及解释性模型如BERT-IRT。此外，文中简要综述了用于CAT中项目参数校准的机器学习方法。
原文摘要 · Abstract (English)
Item response theory (IRT) is a class of interpretable factor models that are widely used in computerized adaptive tests (CATs), such as language proficiency tests. Traditionally, these are fit using parametric mixed effects models on the probability of a test taker getting the correct answer to a test item (i.e., question). Neural net extensions of these models, such as BertIRT, require specialized architectures and parameter tuning. We propose a multistage fitting procedure that is compatible with out-of-the-box Automated Machine Learning (AutoML) tools. It is based on a Monte Carlo EM (MCEM) outer loop with a two stage inner loop, which trains a non-parametric AutoML grade model using item features followed by an item specific parametric model. This greatly accelerates the modeling workflow for scoring tests. We demonstrate its effectiveness by applying it to the Duolingo English Test, a high stakes, online English proficiency test. We show that the resulting model is typically more well calibrated, gets better predictive performance, and more accurate scores than existing methods (non-explanatory IRT models and explanatory IRT models like BERT-IRT). Along the way, we provide a brief survey of machine learning methods for calibration of item parameters for CATs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。