用因果语言建模短路训练，能显著提升生物医学文本的预训练效果。
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
- 先用因果语言建模短期训练，再回退到掩码建模，提升性能。
- 在英法语生物医学任务上，准确率提升1.2-2.8个百分点。
- 适合需要高精度生物医学文本理解的下游任务使用。
将编码器适配新领域时，通常采用掩码语言建模（MLM）继续预训练。本文发现，临时切换至因果语言建模（CLM）并随后进行短时MLM衰减，可显著提升下游性能。在使用ModernBERT处理生物医学文本时，该方法在8个法语和11个英语生物医学任务上优于相同数据与计算量下的标准MLM基线，性能提升达+1.2-2.8个百分点（法语）和+0.3-0.8个百分点（英语），具体取决于模型规模。研究发现，CLM对低层变压器层（0-7）的密集监督作用远超MLM；冻结低层会消除性能增益，而冻结中层则保留增益。表征变化在后续的MLM衰减阶段持续存在，即使衰减阶段长度与CLM阶段一致，且随模型容量增大而增强。本文发布了ModernCamemBERT-bio和ModernBERT-bio，作为当前最优的生物医学编码器，分别适用于Base和Large版本。
原文摘要 · Abstract (English)
When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。