对比从头训练与转换大模型，打造高效德国语编码器。
New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models
- 从零训练现代风格德国编码器ModernGBERT，及转换解码器的LLäMmleinVec
- 1B参数ModernGBERT在SuperGLEBer上达0.808新纪录，超越GBERT Large和7B转换模型
- 适合关注参数效率与延迟的场景，开源全部模型与训练数据
尽管解码器主导的大语言模型兴起，编码器在德国语自然语言处理中仍至关重要。本文在相同数据与训练条件下，比较两种高质量德国编码器构建路径：从零训练与通过LLM2Vec转换解码器。提出两个资源：ModernGBERT（134M, 1B）为完全透明的德语编码器，遵循ModernBERT风格；以及LLäMmleinVec（120M, 1B, 7B），基于掩码下一词预测训练的解码器转编码器模型，上下文长度扩展至8192 tokens。在SuperGLEBer基准上，ModernGBERT 1B达到0.808平均得分，超越GBERT Large（+4%）及七倍大的转换7B模型（0.787）。经监督微调后，在German MTEB上，ModernGBERT 1B（0.551）接近转换7B模型（0.557）。所有模型、检查点、数据集与完整训练记录均公开，引入适配编码器的QA-NIAH评估。结果表明：当参数效率与延迟重要时，从头训练更优；若有预训练解码器且算力受限，转换是有效替代方案。ModernGBERT与LLäMmleinVec及其代码、数据、中间检查点均以研究用途RAIL许可发布。
原文摘要 · Abstract (English)
Encoders remain essential for efficient German NLP and NLU scenarios despite the rise of decoder-only LLMs. This work studies two routes to high-quality German encoders under identical data and training constraints: 1) training from scratch and 2) converting decoders via LLM2Vec. We introduce two resources: ModernGBERT (134M, 1B), fully transparent German encoders in the ModernBERT style, and LLäMmleinVec (120M, 1B, 7B), decoder-to-encoder conversions trained with masked next-token prediction, both undergoing a context extension to 8.192 tokens. Across SuperGLEBer, ModernGBERT 1B sets a new state of the art (avg 0.808), surpassing GBERT Large (+4%) and the seven-times larger converted 7B model (0.787). On German MTEB after supervised fine-tuning, ModernGBERT 1B (0.551) approaches the converted 7B model (0.557). We release all models, checkpoints, datasets, and full training records, and introduce an encoder-adapted QA-NIAH evaluation. All in all, our results provide actionable guidance: when parameter efficiency and latency matter, from-scratch encoders dominate. When a pre-trained decoder exists and compute is a limited, conversion offers an effective alternative. ModernGBERT and LLäMmleinVec, including all code, data and intermediary checkpoints are published under a research-only RAIL license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。