在法律领域对ModernBERT进行微调，显著提升其文本理解能力。
Legal Domain Adaptation of Modern BERT Models
- 用美国法院判决文本继续预训练ModernBERT，增强法律领域适应性。
- 相比原始模型，在所有美国法院数据集上均实现显著性能提升。
- 支持8192词长序列，适用于法律段落嵌入与快速检索排序。
我们研究了现代BERT模型在法律领域的领域适配问题。在全部美国法院判决文本上，使用掩码语言建模目标对ModernBERT进行了进一步预训练。尽管ModernBERT已基于约500倍于原始BERT的数据训练，但我们在法律领域仍发现其通过进一步预训练和领域适配可获得显著提升：在所有与美国法院判决相关的数据集上，性能均优于原版ModernBERT。提升幅度与早期BERT类模型领域适配工作相当。但在实验中，从零开始预训练的效果不及对现有ModernBERT检查点的继续预训练。所获模型可处理最长8,192个标记的序列，能生成有意义的法律段落嵌入，或对数百个法律条文进行快速重排序以响应查询。所有模型检查点均已公开发布。
原文摘要 · Abstract (English)
We investigate domain adaptation of modern BERT models in the legal domain. We further pre-train ModernBERT on all US court opinions using the masked language modeling objective. Although ModernBERT has been trained on roughly 500x more data than original BERT, we still find that this model benefits from further pre-training and domain adaptation in the legal domain: we report significant improvements compared to vanilla ModernBERT on all datasets connected to US court opinions. We find gains similar to those reported in early work on domain adaptation of BERT-like models. However, from scratch pre-training does not match the performance of further pre-training an existing ModernBERT checkpoint in our experiments. The resulting models are capable of processing sequences up to 8,192 tokens, and can be used to compute meaningful embeddings of legal passages, or could quickly rerank hundreds of legal passages for a given search query. We release all model checkpoints publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。