用主动学习仅标注10%数据就达到顶尖模型效果,大幅减少人工标注负担。
Active Learning to Guide Labeling Efforts for Question Difficulty Estimation
- 基于主动学习,每次选最有价值的样本让专家标注。
- 仅用10%标注数据,性能接近全监督模型。
- 适合需要少标注却高精度的教育类应用。
近年来,自然语言处理技术推动了问题难度估计(QDE)研究的发展。基于Transformer的神经网络在监督学习框架下表现优异,但对大量标注数据依赖严重;而无监督方法虽无需标注,却评估成本高且机制不同。本文提出一种主动学习框架,通过迭代训练并仅对最不确定的样本请求人工标注,显著降低标注成本。我们设计了一种新的采样函数PowerVariance,作为分类中PowerBALD的回归扩展,结合DistilBERT与蒙特卡洛丢弃法捕捉未标注样本中的认知不确定性。实验表明,采用PowerVariance的主动学习在仅标注10%训练数据时,性能接近全监督模型。该方法有助于合理使用教育资源,使课程教师更易获取QDE工具,并适用于个性化辅导系统等场景。
原文摘要 · Abstract (English)
In recent years, there has been a surge in research on Question Difficulty Estimation (QDE) using natural language processing techniques. Transformer-based neural networks achieve state-of-the-art performance, primarily through supervised methods but with an isolated study in unsupervised learning. While supervised methods focus on predictive performance, they require abundant labeled data. On the other hand, unsupervised methods do not require labeled data but rely on a different evaluation metric that is also computationally expensive in practice. This work bridges the research gap by exploring active learning for QDE, a supervised human-in-the-loop approach striving to minimize the labeling efforts while matching the performance of state-of-the-art models. The active learning process iteratively trains on a labeled subset, acquiring labels from human experts only for the most informative unlabeled data points. Furthermore, we propose a novel acquisition function PowerVariance to add the most informative samples to the labeled set, a regression extension to the PowerBALD function popular in classification. We employ DistilBERT for QDE and identify informative samples by applying Monte Carlo dropout to capture epistemic uncertainty in unlabeled samples. The experiments demonstrate that active learning with PowerVariance acquisition achieves a performance close to fully supervised models after labeling only 10% of the training data. The proposed methodology promotes the responsible use of educational resources, makes QDE tools more accessible to course instructors, and is promising for other applications such as personalized support systems and question-answering tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。