fine-tuning LLMs做生物医学术语标准化，效果取决于标识符是否常见且易记。
From Memorization to Generalization: Fine-Tuning Large Language Models for Biomedical Term-to-Identifier Normalization
- 根据标识符常见度和可读性设计微调策略，影响模型记忆与泛化能力。
- 基因相关术语微调后准确率提升13.9%，而GO/HPO术语几乎无改善。
- 适合关注生物医学信息整合的研究者或临床数据标准化实践者。
有效的生物医学数据整合依赖于术语标准化，即把自然语言术语映射到标准标识符。这一过程对语义互操作性至关重要。大型语言模型（LLMs）在该任务上展现潜力，但在不同术语体系中表现不一。我们评估了多个生物医学本体库中模型的记憶能力（训练术语表现）与泛化能力（验证术语表现）。对Llama 3.1 8B进行微调发现：GO术语的记憶能力显著提升（准确率最高提高77%），而HPO仅略有改善；只有蛋白-基因（GENE）映射实现了泛化（准确率提升13.9%），其余均无转移效果。基线表现随模型规模变化，GPT-4o在所有术语体系中均优于两个Llama变体。嵌入分析显示：基因符号与蛋白名称间语义对齐紧密，而GO与HPO的术语与标识符间对齐弱，反映出词汇化程度低。微调成功取决于两个相互作用因素：标识符流行度与词汇化水平。流行标识符更常出现在预训练中，利于记忆；可词汇化的标识符（如基因符号）支持语义泛化；而GO与HPO中的任意标识符限制模型只能死记硬背。这些发现为判断微调能否提升事实召回提供了预测框架。
原文摘要 · Abstract (English)
Effective biomedical data integration depends on automated term normalization, the mapping of natural language biomedical terms to standardized identifiers. This linking of terms to identifiers is essential for semantic interoperability. Large language models (LLMs) show promise for this task but perform unevenly across terminologies. We evaluated both memorization (training-term performance) and generalization (validation-term performance) across multiple biomedical ontologies. Fine-tuning Llama 3.1 8B revealed marked differences by terminology. GO mappings showed strong memorization gains (up to 77% improvement in term-to-identifier accuracy), whereas HPO showed minimal improvement. Generalization occurred only for protein-gene (GENE) mappings (13.9% gain), while fine-tuning for HPO and GO yielded negligible transfer. Baseline accuracy varied by model scale, with GPT-4o outperforming both Llama variants for all terminologies. Embedding analyses showed tight semantic alignment between gene symbols and protein names but weak alignment between terms and identifiers for GO or HPO, consistent with limited lexicalization. Fine-tuning success depended on two interacting factors: identifier popularity and lexicalization. Popular identifiers were more likely encountered during pretraining, enhancing memorization. Lexicalized identifiers, such as gene symbols, enabled semantic generalization. By contrast, arbitrary identifiers in GO and HPO constrained models to rote learning. These findings provide a predictive framework for when fine-tuning enhances factual recall versus when it fails due to sparse or non-lexicalized identifiers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。