对比ModernBERT与DeBERTaV3，发现架构改进有限，数据质量影响更大。
ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance
- 在相同数据下重训ModernBERT，隔离架构影响
- DeBERTaV3样本效率更高，整体表现更优
- 模型长上下文支持和训练速度是主要优势
Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT引入了架构改进以提升效率与性能。尽管ModernBERT作者报告其在多个基准上优于DeBERTaV3，但因未公开训练数据且缺乏共享数据集对比，难以判断性能提升源于架构还是数据差异。本文通过在CamemBERTaV2相同的语料上预训练ModernBERT，隔离模型设计影响。结果表明，旧版模型在样本效率和整体性能上仍占优，ModernBERT主要优势在于支持长上下文、训练与推理更快。此外，高质量预训练数据虽加速收敛，但对最终性能提升有限，暗示基准可能存在饱和。研究强调评估Transformer模型时需区分预训练数据与架构创新的影响。
原文摘要 · Abstract (English)
Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT introduce architectural advancements aimed at improving efficiency and performance. Although the authors of ModernBERT report improved performance over DeBERTaV3 on several benchmarks, the lack of disclosed training data and the absence of comparisons using a shared dataset make it difficult to determine whether these gains are due to architectural improvements or differences in training data. In this work, we conduct a controlled study by pretraining ModernBERT on the same dataset as CamemBERTaV2, a DeBERTaV3 French model, isolating the effect of model design. Our results show that the previous model generation remains superior in sample efficiency and overall benchmark performance, with ModernBERT's primary advantage being its support for long context, faster training, and inference speed. However, the new proposed model still provides meaningful architectural improvements compared to earlier models such as BERT and RoBERTa. Additionally, we observe that high-quality pre-training data accelerates convergence but does not significantly improve final performance, suggesting potential benchmark saturation. These findings show the importance of disentangling pretraining data from architectural innovations when evaluating transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。