通过权重合并提升领域检索效果，无需微调且数据少时更优
Effect of Model Merging in Domain-Specific Ad-hoc Retrieval
- 用线性插值融合通用与领域模型权重
- 在医疗和日文领域均优于源模型，有限数据下接近微调效果
- 适合数据稀缺场景，替代LoRA微调的轻量方案
本研究评估了模型合并技术在即席检索任务中的效果。模型合并能融合多个模型的特性，我们假设将其应用于领域特定即席检索可提升检索效果。为此，采用线性插值方法合并源检索模型与领域特定（非检索）模型的权重，该方法无需额外微调。我们在医学和日文两个领域各开展两项实验：第一项比较合并模型与源检索模型；第二项在全量和有限数据设置下，比较合并模型与LoRA微调模型。结果表明，模型合并能生成比源模型更有效的领域检索模型，在数据有限时甚至可作为LoRA微调的实用替代方案。
原文摘要 · Abstract (English)
In this study, we evaluate the effect of model merging in ad-hoc retrieval tasks. Model merging is a technique that combines the diverse characteristics of multiple models. We hypothesized that applying model merging to domain-specific ad-hoc retrieval tasks could improve retrieval effectiveness. To verify this hypothesis, we merged the weights of a source retrieval model and a domain-specific (non-retrieval) model using a linear interpolation approach. A key advantage of our approach is that it requires no additional fine-tuning of the models. We conducted two experiments each in the medical and Japanese domains. The first compared the merged model with the source retrieval model, and the second compared it with a LoRA fine-tuned model under both full and limited data settings for model construction. The experimental results indicate that model merging has the potential to produce more effective domain-specific retrieval models than the source retrieval model, and may serve as a practical alternative to LoRA fine-tuning, particularly when only a limited amount of data is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。