用医学本体生成教材式文本，提升医疗编码模型性能。
OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
- 将医学本体图谱转为层级因果关系，通过随机游走捕获知识
- 大模型重写为流畅教材文本，训练时同步优化语言建模与编码关系预测
- 在3个法语医疗标注任务上显著提效，适合医疗自然语言处理研究者
我们提出 OntoBook，一种将医学本体结构转化为编码器语言模型预训练信号的方法。该方法分三步：通过本体图谱的随机游走捕捉医学代码间的层次与因果关系；利用大语言模型将这些路径重构成流畅的教材风格文本；最终使用生成文本对 ModernCamemBERT（149M 参数）进行双目标训练，同时优化掩码语言建模和代码对的关系预测。在三个法语医疗编码基准（FRACCO、Cantemist-FR、Distemist-FR）上，相比仅使用掩码语言建模的预训练，OntoBook 在 FRACCO 上提升 2.5 微 F1，Distemist 上提升 8.0 微 F1。我们发现目标对齐至关重要：若两任务使用不同数据，性能下降 30 分。我们发布了基于三个法语本体（CIM-10、CCAM、ATC）生成的 130 万条教材文本及预训练模型检查点。
原文摘要 · Abstract (English)
We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models. Our approach has three stages: random walks through ontology graphs capture hierarchical and causal relations between medical codes, a large language model reformulates these walks into fluent textbook-style prose, and the resulting text is used to train ModernCamemBERT, a 149M-parameter French encoder, with two objectives on the same data: masked language modeling and relation prediction between code pairs. On three French medical coding benchmarks (FRACCO, Cantemist-FR, Distemist-FR), OntoBook achieves significant improvements over MLM-only pretraining, with +2.5 micro-F1 on FRACCO and +8.0 micro-F1 on Distemist. We find that alignment between objectives is necessary: misaligned training, where each task uses different data, causes a 30-point degradation. We release 1.3 million LLM-reformulated medical textbooks across three French ontologies (CIM-10, CCAM, ATC) and pretrained model checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。