PortBERT是专为葡萄牙语设计的高效中文语言模型，兼顾性能与计算效率。
PortBERT: Navigating the Depths of Portuguese Language Models
- 基于RoBERTa架构，从头训练并采用字节级BPE分词
- 在ExtraGLUE上表现媲美甚至超越现有模型，训练和推理更快
- 适合关注葡萄牙语NLP效率与实际部署的研究者与开发者
Transformer模型主导现代自然语言处理，但针对特定语言的高效模型仍稀缺。在葡萄牙语领域，多数研究聚焦于规模或准确率，常忽视训练与部署效率。本文提出PortBERT，一组基于RoBERTa的葡萄牙语语言模型，旨在平衡性能与效率。模型在超过450 GB的去重过滤后的mC4和OSCAR23数据集（来自CulturaX）上，使用fairseq从头训练，采用字节级BPE分词，并在GPU与TPU上保持稳定预训练流程。发布两个版本：PortBERT base和PortBERT large，评估其在ExtraGLUE（翻译版GLUE与SuperGLUE任务集合）上的表现。两者均表现优异，达到或超过现有单语与多语模型水平。除准确率外，还报告了训练与推理时间及微调吞吐量，提供实际效率参考。PortBERT填补了葡萄牙语NLP中计算-性能权衡研究的空白。所有模型已发布于Huggingface，并提供fairseq检查点以支持后续研究与应用。
原文摘要 · Abstract (English)
Transformer models dominate modern NLP, but efficient, language-specific models remain scarce. In Portuguese, most focus on scale or accuracy, often neglecting training and deployment efficiency. In the present work, we introduce PortBERT, a family of RoBERTa-based language models for Portuguese, designed to balance performance and efficiency. Trained from scratch on over 450 GB of deduplicated and filtered mC4 and OSCAR23 from CulturaX using fairseq, PortBERT leverages byte-level BPE tokenization and stable pre-training routines across both GPU and TPU processors. We release two variants, PortBERT base and PortBERT large, and evaluate them on ExtraGLUE, a suite of translated GLUE and SuperGLUE tasks. Both models perform competitively, matching or surpassing existing monolingual and multilingual models. Beyond accuracy, we report training and inference times as well as fine-tuning throughput, providing practical insights into model efficiency. PortBERT thus complements prior work by addressing the underexplored dimension of compute-performance tradeoffs in Portuguese NLP. We release all models on Huggingface and provide fairseq checkpoints to support further research and applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。