用模块化专家合并提升生物医学检索效果，无需大规模混合训练。
Modular Expert Merging for Biomedical Retrieval
- 独立训练的领域专家通过参数空间组合，比混合训练更有效。
- 小模型在合成难例后微调，检索性能提升显著，最大增益达15.2%。
- 框架无需持续预训练，适合医疗和通用检索任务快速部署。
将通用大模型适配为领域专用稠密检索器通常需要在混合数据上进行大规模训练。我们发现，对四个解码器仅有的语言模型家族（0.6B-7B）及四种合并方法，在十二项来自MTEB的医学与通用检索任务中，独立训练的领域专家合并表现持续优于混合训练，表明参数空间组合能捕捉互补领域优势，而大规模混合训练会弱化这些特性。为进一步提升专家质量，我们提出合成-训练-合并（STM）框架：利用顶级大模型生成硬负样本，通过LoRA微调领域专家后再合并，无需持续预训练。合成硬负样本使小模型获益最大，STM在生物医学检索任务中表现优异，同时在所有四个骨干模型上保持了与通用领域相当的竞争力。
原文摘要 · Abstract (English)
Adapting general-purpose LLMs into domain-specialized dense retrievers typically requires large-scale training on mixed-domain data. We show that merging independently trained domain-specialized experts consistently exceeds this approach across four decoder-only LLM families (0.6B-7B), four merging methods, and twelve medical and general retrieval tasks from MTEB, suggesting that parameter-space composition captures complementary domain strengths that large-scale mixed-domain training averages out. To further maximize expert quality, we introduce Synthesize-Train-Merge (STM), a modular framework that synthesizes hard negatives with a top-tier LLM and fine-tunes domain-specialized experts via LoRA before merging them, without continual pre-training. Synthesized hard negatives yield the largest gains for smaller models, and STM achieves strong performance on biomedical retrieval tasks while maintaining competitive general-domain results across all four backbone families.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。