mmBERT用新训练策略提升多语言模型性能，低资源语言也能高效学习。
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
- 采用逆向掩码和温度采样策略，优化多语言训练过程。
- 仅在衰减阶段加入1700+低资源语言，性能接近o3和Gemini 2.5 Pro。
- 适合需要高覆盖多语言能力的研究与应用，尤其低资源场景。
编码器类语言模型广泛应用于分类与检索等任务，但近年来针对多语言编码器的研究较少。本文提出mmBERT，一个在超过1800种语言的3万亿词文本上预训练的编码器模型。创新性地引入逆向掩码比例调度和逆向温度采样比例，仅在训练衰减阶段加入1700余种低资源语言，显著提升性能，并最大化有限数据的收益。尽管低资源语言仅在短时段内参与训练，其分类表现仍与OpenAI o3和Google Gemini 2.5 Pro相当。整体上，mmBERT在高低资源语言的分类与检索任务中均显著优于前代模型。
原文摘要 · Abstract (English)
Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。