TabiBERT是首个从头训练的土耳其语现代BERT模型，性能领先。
TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
- 基于ModernBERT架构，融合旋转位置编码与FlashAttention，支持8192词长上下文
- 在1万亿词上预训练，推理速度提升2.65倍，内存占用降低，支持更大批量
- 在28个任务的统一评测集上表现最优，尤其在问答和学术理解任务中优势明显
自BERT问世以来，仅编码器的Transformer在计算效率、训练稳定性及长文本建模方面取得显著进展。ModernBERT通过集成旋转位置编码（RoPE）、FlashAttention和优化归一化实现这些突破。然而，土耳其语自然语言处理领域仍缺乏一个从头训练、采用现代架构的单语编码器。本文提出TabiBERT，一个基于ModernBERT架构、从头训练的土耳其语单语编码器，使用848.8亿词的多领域语料库中的1万亿词进行预训练：网络文本占73%，科学文献占20%，源代码占6%，数学内容占0.3%。该模型支持8192词的上下文长度（为原始BERT的16倍），推理速度最高提升2.65倍，降低GPU内存消耗，支持更大批量。我们构建了包含28个数据集、涵盖8类任务的统一评测基准TabiBench，采用类似GLUE的宏平均评估。TabiBERT在该基准上取得77.58分，优于BERTurk 1.62分，在8个类别中有5个达到当前最佳，尤其在问答任务（+9.55分）、代码检索（+2.41分）和学术理解（+0.66分）上提升显著。相比以往特定任务的最佳结果（包括TurkishBERTweet等专用模型），平均提升1.47分，表明其出色的跨领域泛化能力。模型权重、训练配置与评估代码均已开源，促进可复现的土耳其语编码器研究。
原文摘要 · Abstract (English)
Since the inception of BERT, encoder-only Transformers have evolved significantly in computational efficiency, training stability, and long-context modeling. ModernBERT consolidates these advances by integrating Rotary Positional Embeddings (RoPE), FlashAttention, and refined normalization. Despite these developments, Turkish NLP lacks a monolingual encoder trained from scratch, incorporating such modern architectural paradigms. This work introduces TabiBERT, a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus. TabiBERT is pre-trained on one trillion tokens sampled from an 84.88B token multi-domain corpus: web text (73%), scientific publications (20%), source code (6%), and mathematical content (0.3%). It supports 8,192-token context length (16x original BERT), achieves up to 2.65x inference speedup, and reduces GPU memory consumption, enabling larger batch sizes. We introduce TabiBench with 28 datasets across eight task categories with standardized splits and protocols, evaluated using GLUE-style macro-averaging. TabiBERT attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories, with particularly strong gains on question answering (+9.55 points), code retrieval (+2.41 points), and academic understanding (+0.66 points). Compared with task-specific prior best results, including specialized models like TurkishBERTweet, TabiBERT achieves +1.47 average improvement, indicating robust cross-domain generalization. We release model weights, training configurations, and evaluation code for transparent, reproducible Turkish encoder research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。