提出分层整合方法,高效融合医疗大模型知识。
A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs
- 分层整合:用最优传输对齐注意力层,再加余弦加权插值。
- 简单平均法在医学问答上达45.80%准确率,优于复杂剪枝方法。
- 适合资源受限的医疗边缘部署,兼顾效率与模型兼容性。
大型语言模型在分布式医疗中面临挑战:如何在保护隐私、降低计算开销的同时,整合跨机构的专业知识,并避免灾难性遗忘。本文系统评估了六种参数空间融合技术在两个基于Mistral-7B架构的医学大模型上的表现。提出一种新型分层方法,结合选择性最优传输(OT)对齐注意力层与余弦相似度加权插值,以应对排列方差问题,同时降低边缘部署的计算开销。在五个医学基准测试中对比了任务算术、线性平均、DARE-TIES、DELLA、Breadcrumbs及本方法。结果表明,架构兼容的模型通过简单平均即可显著受益,任务算术在MedQA上达到45.80%准确率,优于复杂的剪枝方法。研究为资源受限的物联网医疗环境中的分布式医疗AI部署提供了关键洞见,证实对于架构兼容模型,简单平均是知识融合的稳健且高效的基线方案,为可扩展医疗AI系统提供可行路径。
原文摘要 · Abstract (English)
Large Language Models (LLMs) face significant challenges in distributed healthcare, including consolidating specialized domain knowledge across institutions while maintaining privacy, reducing computational overhead, and preventing catastrophic forgetting during model updates.This paper presents a systematic evaluation of six parameter-space merging techniques applied to two architecturally compatible medical LLMs derived from the Mistral-7B base model. We introduce a novel hierarchical method that combines selective Optimal Transport (OT) alignment for attention layers with cosine similarity-weighted interpolation, designed to address permutation variance while minimizing computational overhead for edge deployment scenarios. Our study evaluates Task Arithmetic, Linear Averaging, DARE-TIES, DELLA, Breadcrumbs, and our Hierarchical approach across five medical benchmarks. Results demonstrate that architecturally compatible models benefit significantly from simple averaging methods, with Task Arithmetic achieving 45.80% accuracy on MedQA, outperforming complex pruning-based approaches. These findings offer critical insights for the deployment of distributed medical AI in resource-constrained IoT environments, where computational efficiency and model compatibility are paramount. Our work establishes that for architecturally compatible models, simple averaging provides a robust and computationally efficient baseline for knowledge consolidation, offering a pragmatic path forward for scalable medical AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。