ModernBERT提升长文本编码效率，兼顾速度、内存与性能。
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
- 引入现代优化技术重构编码器，实现高效长序列处理
- 在2万亿词上训练，8192序列长度下达到顶尖分类与检索效果
- 适配普通GPU推理，速度与内存效率均领先现有编码器
如BERT等仅编码器的Transformer模型在检索与分类任务中相比更大规模的解码器模型，具有出色的性能-大小权衡。尽管广泛应用于众多生产流水线，自BERT发布以来其性能提升有限。本文提出ModernBERT，将现代模型优化技术引入仅编码器架构，实现了对旧版编码器的重大帕累托改进。该模型在2万亿词数据上以原生8192序列长度进行训练，在涵盖多样化分类任务及不同领域（包括代码）的单向量与多向量检索评估中表现优异。除下游任务性能强外，ModernBERT也是目前最快速、最节省内存的编码器，专为常见GPU上的推理设计。
原文摘要 · Abstract (English)
Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。