针对中文特性优化的BERT模型,提升长文本处理与检索能力。
Chinese ModernBERT with Whole-Word Masking
- 自研32k词表+全词掩码,适配中文构词特点降低嵌入开销
- 支持8192长度上下文,长序列吞吐高,短序列速度不降
- 加入对比数据微调后在检索任务上超越0.6B小模型
编码器类Transformer在架构、数据和系统三方面持续进步,但在中文领域因分词与形态差异未充分受益。本文提出Chinese ModernBERT,从零构建中文编码器:(i) 设计32k BPE词表,专用于高频中文词缀/复合词,减少嵌入预算;(ii) 采用动态掩码率(30%→15%)的全词掩码策略,随训练进度调整任务难度;(iii) 两阶段预训练流程,结合RoPE与交替局部/全局注意力,将原生上下文扩展至8,192个词元;(iv) 使用阻尼余弦学习率调度,实现长期优化稳定。模型在约1.2万亿中文字符数据(CCI3-HQ、CCI4、Cosmopedia-Chinese)上预训练。在CLUE基准上表现媲美强基线中文编码器。启用bf16后兼具高长序列吞吐与良好短序列速度。引入少量开放对比数据(SimCLUE,约300万对),再叠加T2Ranking(约200万对)微调后,在SimCLUE测试集上达到0.505(皮尔逊)/0.537(斯皮尔曼)得分,超越Qwen-0.6B-embedding,表明基于更多精选对的可扩展路径。代码与权重将公开以支持复现。
原文摘要 · Abstract (English)
Encoder-only Transformers have advanced along three axes -- architecture, data, and systems -- yielding Pareto gains in accuracy, speed, and memory efficiency. Yet these improvements have not fully transferred to Chinese, where tokenization and morphology differ markedly from English. We introduce Chinese ModernBERT, a from-scratch Chinese encoder that couples: (i) a hardware-aware 32k BPE vocabulary tailored to frequent Chinese affixes/compounds, lowering the embedding budget; (ii) whole-word masking (WWM) with a dynamic masking curriculum (30% -> 15%) to align task difficulty with training progress; (iii) a two-stage pre-training pipeline that extends the native context from 1,024 to 8,192 tokens using RoPE and alternating local/global attention; and (iv) a damped-cosine learning-rate schedule for stable long-horizon optimization. We pre-train on ~1.2T Chinese tokens from CCI3-HQ, CCI4 (Chinese), and Cosmopedia-Chinese. On CLUE, Chinese ModernBERT is competitive with strong Chinese encoders under a unified fine-tuning protocol. Under bf16 it achieves high long-sequence throughput while maintaining strong short-sequence speed, reflecting benefits from budget allocation and attention design. To probe retrieval-oriented quality, we add a small amount of open contrastive data: fine-tuning on SimCLUE (~3M pairs) improves further when adding T2Ranking (~2M), reaching 0.505 (Pearson) / 0.537 (Spearman) on the SimCLUE test set. Under this open-data setting, Chinese ModernBERT surpasses Qwen-0.6B-embedding on SimCLUE, suggesting a clear scaling path for STS with additional curated pairs. We will release tokenizer and weights to facilitate reproducible research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。