根据文本长度动态调整正则强度，提升模型预测可靠性。
Confidence Regularized Masked Language Modeling using Text Length
- 按输入长度自适应调节置信度正则化强度
- 在GLUE和SQuAD上同时提升准确率与校准误差
- 特别适合短文本场景下的语言模型训练
掩码语言建模是学习语言表征的常用方法，模型需预测输入中随机掩码的词。然而，该方法通常仅考虑单一正确答案，忽略了人类可能选择的多种合理替代项。当输入文本较短时，词汇分布熵更高，模型易产生过度自信。为此，我们提出一种新型置信度正则化方法，依据输入长度自适应调整正则强度。在GLUE和SQuAD基准上的实验表明，该方法在准确率与预期校准误差方面均有提升。
原文摘要 · Abstract (English)
Masked language modeling is a widely used method for learning language representations, where the model predicts a randomly masked word in each input. However, this approach typically considers only a single correct answer during training, ignoring the variety of plausible alternatives that humans might choose. This issue becomes more pronounced when the input text is short, as the possible word distribution tends to have higher entropy, potentially causing the model to become overconfident in its predictions. To mitigate this, we propose a novel confidence regularizer that adaptively adjusts the regularization strength based on the input length. Experiments on the GLUE and SQuAD benchmarks show that our method improves both accuracy and expected calibration error
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。