改进掩码语言模型，提升小规模预训练模型性能
Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
- 根据模型预测能力动态调整掩码概率，优化掩码策略
- 在(Super)GLUE任务上显著超越标准MLM，性能大幅提升
- 引入子词嵌入，增强模型对词形变化的泛化能力
本文介绍2025年BabyLM挑战赛的参赛策略。主要贡献是提出一种改进的掩码语言模型（MLM）方法，根据模型对被掩码词元的预测能力动态调整其掩码概率。实验结果表明，该方法在(Super)GLUE任务上的表现显著优于标准MLM。此外，我们引入子词嵌入，进一步提升了模型对形态学变化的泛化能力。我们的提交在严格小规模赛道中击败了基线模型。
原文摘要 · Abstract (English)
We describe our strategy for the 2025 edition of the BabyLM Challenge. Our main contribution is that of an improved form of Masked Language Modeling (MLM), which adapts the probabilities of the tokens masked according to the model's ability to predict them. The results show a substantial increase in performance on (Super)GLUE tasks over the standard MLM. We also incorporate sub-token embeddings, finding that this increases the model's morphological generalization capabilities. Our submission beats the baseline in the strict-small track.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。