用熵值选要掩码的词，让语言模型学得更高效
Entropy-aware Masking for Masked Language Modeling
- 根据预测熵值选择高不确定性词进行掩码
- 在GLUE上平均提升5%得分，优于随机掩码
- 无需外部模型，自掩码设计适合大规模训练
掩码语言建模已成为编码器类语言模型预训练的标准目标。该方法通过掩码输入中的部分词元，并利用上下文信息让模型预测被掩码内容，从而捕捉语言的句法与语义特性。传统方法随机选择掩码词元，可能无法提供最优学习信号。本文提出基于熵分布的掩码策略，利用模型对词元预测的熵值识别更具信息量和不确定性的词元进行掩码，以提升训练效率。此外，我们提出一种新颖的自掩码方法，在不依赖外部参考模型的前提下增强训练效率。实验表明，该方法在GLUE基准上平均性能提升5%，结合知识蒸馏后取得最佳结果。
原文摘要 · Abstract (English)
Masked language modeling has become a standard pretraining objective for training encoder-based language models. In this approach, certain tokens in the input are masked, and the model learns to predict them using the surrounding context. This process enables the model to capture both syntactic and semantic properties of language. Conventionally, the tokens selected for masking are chosen at random, which may not always yield the most effective learning signals. In this work, we examine a token masking strategy based on entropy distribution. We use the model's entropy over token predictions to identify which tokens should be masked. This method aims to target tokens that are more informative and uncertain to improve the training efficacy. We also propose a novel self-masking approach that enhances training efficiency without relying on an external reference model. Experimental results demonstrate that our method achieves an average performance improvement of 5% in GLUE scores compared to the baseline. Further, we experiment with combining knowledge distillation with entropy masking, resulting in the best overall results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。