研究噪声数据对BERT在句子难度检测中的影响，发现去噪可显著提升小数据集性能。
How Much Noise Can BERT Handle? Insights from Multilingual Sentence Difficulty Detection
- 用多种去噪方法（如GMM、Co-Teaching）处理众包获取的噪声标注数据
- 小数据集上去噪使AUC从0.52提升至0.93，大模型表现已接近上限
- 发布最大多语言句子难度数据集，适合低资源语言研究者使用
噪声训练数据会显著降低基于语言模型分类器的性能，尤其在非主题类任务中。本文设计方法框架评估去噪影响，探索了多种句级难度检测的去噪策略，基于通过噪声众包获得的文档级难度标注数据。涵盖单语与跨语言迁移场景，即多语言模型在一种语言训练后在另一种语言测试。评估了高斯混合模型（GMM）、Co-Teaching、噪声转移矩阵和标签平滑等去噪技术。结果表明，尽管BERT模型本身具备抗噪能力，但显式去噪仍可进一步提升性能。在较小数据集上，基于GMM的去噪将AUC从0.52提升至0.92，结合多种方法可达0.93；而在较大数据集上，预训练模型的内在正则化已提供强基线，去噪仅带来微弱提升（从0.92到0.94），两种方法组合无增益。然而，移除约20%的噪声句子有助于构建更清洁的语料库。因此，本研究发布了目前最大的多语言句子难度预测语料库：https://github.com/Nouran-Khallaf/denoising-difficulty
原文摘要 · Abstract (English)
Noisy training data can significantly degrade the performance of language-model-based classifiers, particularly in non-topical classification tasks. In this study we designed a methodological framework to assess the impact of denoising. More specifically, we explored a range of denoising strategies for sentence-level difficulty detection, using training data derived from document-level difficulty annotations obtained through noisy crowdsourcing. Beyond monolingual settings, we also address cross-lingual transfer, where a multilingual language model is trained in one language and tested in another. We evaluate several noise reduction techniques, including Gaussian Mixture Models (GMM), Co-Teaching, Noise Transition Matrices, and Label Smoothing. Our results indicate that while BERT-based models exhibit inherent robustness to noise, incorporating explicit noise detection can further enhance performance. For our smaller dataset, GMM-based noise filtering proves particularly effective in improving prediction quality by raising the Area-Under-the-Curve score from 0.52 to 0.92, or to 0.93 when de-noising methods are combined. However, for our larger dataset, the intrinsic regularisation of pre-trained language models provides a strong baseline, with denoising methods yielding only marginal gains (from 0.92 to 0.94, while a combination of two denoising methods made no contribution). Nonetheless, removing noisy sentences (about 20\% of the dataset) helps in producing a cleaner corpus with fewer infelicities. As a result we have released the largest multilingual corpus for sentence difficulty prediction: see https://github.com/Nouran-Khallaf/denoising-difficulty
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。