训练出首个支持8192长上下文的日本语BERT模型，推动中文之外的东亚语言预训练发展。
llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length
- 基于超大规模日语文本，构建支持8192词元长上下文的ModernBERT模型。
- 在填空测试中表现良好，但下游任务性能未超越现有基线。
- 开源模型与代码，助力长上下文BERT研究与复现。
如BERT等仅编码器的Transformer模型广泛用于句子分类与检索任务。然而，相比解码器主导的模型，大规模语料与长上下文预训练的探索仍不足。本文提出llm-jp-modernbert，一个在公开大规模日语文本上训练、上下文长度达8192词元的ModernBERT模型。尽管其在下游任务上未超越现有基线，但在填空掩码测试中表现良好。我们通过伪困惑度实验分析了上下文长度扩展的影响。此外，详细研究了句子嵌入在训练中的演变，并与其他同架构模型对比，确认趋势一致。为促进可复现性及长上下文BERT的发展，我们发布模型及其训练与评估代码。
原文摘要 · Abstract (English)
Encoder-only transformer models like BERT are widely adopted as a pre-trained backbone for tasks like sentence classification and retrieval. However, pretraining of encoder models with large-scale corpora and long contexts has been relatively underexplored compared to decoder-only transformers. In this work, we present llm-jp-modernbert, a ModernBERT model trained on a publicly available, massive Japanese corpus with a context length of 8192 tokens. While our model does not surpass existing baselines on downstream tasks, it achieves good results on fill-mask test evaluations. We also analyze the effect of context length expansion through pseudo-perplexity experiments. Furthermore, we investigate sentence embeddings in detail, analyzing their transitions during training and comparing them with those from other existing models, confirming similar trends with models sharing the same architecture. To support reproducibility and foster the development of long-context BERT, we release our model, along with the training and evaluation code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。