专为葡萄牙语优化的新型编码器，性能超越现有模型。
BERTomelo: Your Portuguese Encoder Best Friend
- 基于ModernBERT架构从零训练，支持1024词长上下文。
- 在STS和NER任务上优于BERTimbau等旧模型，效率更高。
- 适合需要高精度葡萄牙语理解的应用场景。
编码器已成为多项自然语言处理任务的主流方法，尤其在需要深层语境理解的任务中表现突出。尽管多语言模型覆盖广泛，但针对特定语言的单语编码器对捕捉语言独有的词汇与语法特征至关重要。对于葡萄牙语而言，现有单语模型如BERTimbau和Albertina未能跟上最新架构进展，在可扩展性和效率方面普遍落后于英语基准。本文提出BERTomelo，一种从头预训练的下一代单语编码器，专为葡萄牙语优化。该模型采用ModernBERT架构，提供Base和Large版本，支持1,024词的上下文窗口，并集成FlashAttention与交替注意力机制等硬件级优化。模型在包含1.06亿文档的高质量葡萄牙语语料库ClassiCC-PT上训练，确保与现代葡萄牙语用法高度一致。实验表明，BERTomelo不仅全面超越先前葡萄牙语编码器，还在下游任务（如语义文本相似度和命名实体识别）中提供了比大型多语言模型更稳健、高效的替代方案。
原文摘要 · Abstract (English)
Encoders have become the state of the art for multiple NLP tasks, especially those requiring deep contextual understanding. While multilingual models offer broad coverage, dedicated monolingual encoders are essential for capturing the unique lexical and syntactic nuances of specific languages. For Portuguese, however, existing monolingual options like BERTimbau and Albertina have not kept pace with recent architectural breakthroughs, often lagging behind English benchmarks in scalability and efficiency. This work introduces BERTomelo, a next-generation monolingual encoder pre-trained from scratch and specifically optimized for the Portuguese language. By leveraging the ModernBERT architecture, BERTomelo overcomes the limitations of previous models, offering Base and Large versions with a 1,024-token context window and hardware-level optimizations like FlashAttention and alternating attention mechanisms. The model was trained on ClassiCC-PT, a massive, high-quality Portuguese corpus of 106 million documents, ensuring superior alignment with the language's contemporary usage. The results demonstrate that BERTomelo not only outperforms previous Portuguese encoders but also provides a more robust and efficient alternative to massive multilingual models in downstream tasks such as STS and NER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。