用渐进式训练法打造更强的土耳其语基础模型，性能超越同类
MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum
- 先用自回归建模后转为掩码建模，提升语言表征能力
- 在土耳其语检索与理解任务中表现最佳，最高领先3.62分
- 适合需要高质量土耳其语表示的NLP研究与应用开发者
土耳其编码器模型虽采用现代架构，但预训练目标仍固定为掩码语言建模。本文提出MoganBert-TR，一个从零训练的149M参数土耳其语编码器基础模型，基于语言特异性过滤语料库训练，共使用2373亿个词元。采用两阶段CLM-to-MLM课程：先进行因果语言建模，再转入掩码语言建模，过渡点设在WSD调度的稳定阶段。在同等训练步数下，该设计在土耳其MS MARCO检索任务上相比纯MLM提升2.7–3.7倍；其机制源于嵌入空间几何，纯MLM下单一方向解释28.1%方差，而课程学习仅11.9%。长上下文扩展与学习率衰减分离为双分支，最后部分以1024上下文运行，使TrGLUE平均提升0.49±0.26分（五组种子，p=0.013），比模型合集方案高0.75分，成本仅增加4.3%。MoganBert-TR在TrGLUE得分为78.41，优于所有对比的土耳其ModernBERT模型；在TabiBench得77.73，代码检索类领先3.62分。衍生嵌入模型MoganBert-Embed通过教师蒸馏与多信号对比微调，在MTEB(Turkish)总体平均达68.30，达到7.57B参数教师模型的99.5%，且模型规模缩小51倍。配套50,048词元分词器在压缩率和词汇丰富度上均优于所有对比土耳其分词器。
原文摘要 · Abstract (English)
Turkish encoder models have adopted modern architectures while leaving the pretraining objective fixed at masked language modelling. This paper introduces MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch on a language-specifically filtered corpus, together with an embedding model derived from it (MoganBert-Embed). MoganBert-TR is trained over 237.3B tokens with a two-stage CLM-to-MLM curriculum: causal language modelling first, masked language modelling for the remainder, with the transition made inside the stable phase of a WSD schedule. In a controlled ablation under an equal step budget, this design outperforms pure MLM by 2.7-3.7x on Turkish MS MARCO retrieval; the measured mechanism is embedding geometry, where a single direction absorbs 28.1% of the variance under pure MLM against 11.9% under the curriculum. Long-context extension and learning-rate decay are then split into two branches after a shared prefix: running the final portion of decay at 1024 context improves the TrGLUE average by 0.49 +/- 0.26 points across five paired seeds (p = 0.013) and beats a model-soup alternative by 0.75 points at ~4.3% additional cost. MoganBert-TR attains 78.41 on TrGLUE, the best among the Turkish ModernBERT models compared, and 77.73 on TabiBench, where it leads two of the eight categories with the largest margin on code retrieval (+3.62 points over TabiBERT). MoganBert-Embed, produced through teacher distillation and multi-signal contrastive fine-tuning, ranks first among student models on the MTEB(Turkish) overall average with 68.30 and reaches 99.5% of its 7.57B-parameter teacher's score with a 51x smaller backbone. The accompanying 50,048-token tokenizer outperforms all compared Turkish tokenizers on compression and fertility across two independent test sets. Weights, tokenizer, embedding model and evaluation code: https://huggingface.co/moganai
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。