用网页数据提升医学编码器预训练效果，让模型更懂专业术语。
Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining
- 通过医学术语密度筛选和大模型重写增强信号，优化网页数据
- 在法语医疗任务中，新方法显著优于传统教育质量过滤
- 适用于法语医疗NLP研究者，尤其关注临床实体识别场景
网页数据筛选在解码器大语言模型预训练中已被广泛研究，而医学等密集术语领域中的编码器预训练仍依赖小规模人工标注语料，限制了可扩展性和写作风格多样性，非英语临床场景下尤为严峻。本研究探讨大规模网页数据对密集术语领域编码器掩码语言建模的适用性。提出两个互补策略：医学术语密度过滤，筛选高医学术语含量文档；信号增强重写，使用大模型将原文改写为术语更密集、实体上下文更广的版本。在法语医疗NLP任务上验证，术语密度过滤优于常用教育质量过滤，二者互补。仅使用重写数据即优于原始网页数据，结合过滤与重写数据可实现最大提升。最终构建了法语医疗预训练语料FineMed及对应模型DoctoBERT，其在公开基准DrBenchmark和私有临床命名实体识别任务上均达当前最佳水平。
原文摘要 · Abstract (English)
Web data curation has been widely studied for decoder Large Language Model (LLM) pretraining. Encoders for dense-terminology domains such as medicine, by contrast, are pretrained on small, manually-curated corpora that limit scalability and writing style diversity, a bottleneck even more severe in non-English clinical settings. Whether web-scale data curation also benefits encoder Masked Language Modeling (MLM) in a dense-terminology domain remains an open question. To address this, we introduce two complementary levers. Medical-term density filtering selects documents rich in medical terms. Signal-amplifying rephrasing uses an LLM to rewrite documents into denser variants with broader entity contexts. We instantiate the recipe on French medical NLP. The medical-term density filter outperforms the widely-used educational quality filter on downstream medical tasks, and the two complement each other. Signal-amplifying rephrasing alone improves on raw web data, and mixing it with filtered web data produces the largest gain. The recipe yields FineMed, a French medical pretraining corpus, and DoctoBERT, a state-of-the-art French medical encoder family evaluated on both the public benchmark DrBenchmark and a proprietary clinical Named Entity Recognition (NER) task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。