用微调Transformer直接预测选择题难度,无需答题数据。
Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning
- 端到端微调Transformer编码题目文字,避免人工特征工程。
- 多任务学习版本在小样本下显著提升预测准确率。
- 适合心理测量学研究者或教育评估系统开发者使用。
无回答项难度建模有望减少对答题数据的依赖,但在阅读理解类选择题中因推理需求分布于词汇组件而极具挑战。现有方法多提取题干特征后交由独立模型处理,我们则通过端到端微调Transformer编码器直接解析题干文本,省去手工特征工程与预处理中的信息损失。进一步提出两种改进:组件级编码法(共享编码器分别处理题干组件)与多任务法(在共享编码器上加入辅助问答任务)。三种训练集规模下均采用蒙特卡洛子采样设计评估。结果表明,联合编码是可行替代方案;组件级方法未见显著优势,与自注意力机制已捕获跨组件关联一致;多任务法在最小样本下表现显著更优。说明经合适辅助任务正则化的变压器可有效恢复典型实际测量场景下的语义可推导信号。该框架支持心理测量学导向的定制扩展。
原文摘要 · Abstract (English)
Response-free item difficulty modelling promises to reduce reliance on response-based calibration but is intrinsically difficult on reading-comprehension multiple-choice items, where difficulty depends on inferential demands across wording components. Whereas most existing approaches extract item-text features and pass them to a separate statistical or machine-learning model, we fine-tune transformer encoders end-to-end on the item wording, eliminating the manual feature engineering and preprocessing that discards information. Moreover, two extensions to this joint-encoding approach are proposed: a component-wise variant that encodes wording components separately through a shared encoder, and a multi-task variant that retains joint encoding and adds an auxiliary multiple-choice question answering objective on the shared encoder. Each method is evaluated under a Monte Carlo subsampling design at three training-set sizes on a held-out test set. We find that joint encoding is a viable end-to-end alternative to feature-engineering pipelines; while the component-wise variant shows no detectable benefit, consistent with self-attention already harvesting the cross-component signal, the multi-task variant delivers significant paired improvements in the smallest-sample regime. Transformer fine-tuning, especially if regularised by a suitable auxiliary task, recovers a substantial share of the wording-derivable signal at training-set sizes typical of applied measurement. The framework provides a customisable interface for psychometrically motivated extensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。