专为医学文本设计的长序列编码器，支持8192词元上下文。
Clinical ModernBERT: An efficient and long context encoder for biomedical text
- 基于ModernBERT架构，引入旋转位置编码与闪速注意力机制。
- 在临床NLP基准上表现优异，长文本表征能力显著提升。
- 适合需要处理长篇病历或医学文献的研究者使用。
我们提出Clinical ModernBERT，一种基于Transformer的编码器，在大规模生物医学文献、临床记录和医学术语库上预训练，融合了PubMed摘要、MIMIC-IV临床数据及医学代码与其文本描述。该模型基于当前最先进的Natural Language Text Encoder ModernBERT，采用旋转位置编码（RoPE）、闪速注意力（Flash Attention）等架构升级，并将上下文长度扩展至8,192个词元。Clinical ModernBERT在生成语义丰富的长上下文表征方面表现出色。我们通过分析预训练权重并结合全面的临床自然语言处理基准测试，验证了其有效性。
原文摘要 · Abstract (English)
We introduce Clinical ModernBERT, a transformer based encoder pretrained on large scale biomedical literature, clinical notes, and medical ontologies, incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with their textual descriptions. Building on ModernBERT the current state of the art natural language text encoder featuring architectural upgrades such as rotary positional embeddings (RoPE), Flash Attention, and extended context length up to 8,192 tokens our model adapts these innovations specifically for biomedical and clinical domains. Clinical ModernBERT excels at producing semantically rich representations tailored for long context tasks. We validate this both by analyzing its pretrained weights and through empirical evaluation on a comprehensive suite of clinical NLP benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。