用数学运算合并大模型权重，实现零样本检索性能提升
Investigating Task Arithmetic for Zero-Shot Information Retrieval
- 通过加减模型权重融合多任务知识，无需微调即可适应新任务
- 在科学、生物医学等数据集上，NDCG@10提升18%，P@10提升15%
- 适合需要快速部署、无标注数据的零样本检索场景
大型语言模型在多种自然语言处理任务中展现出出色的零样本性能，包括文档重排序。然而，其在未见任务和领域中的表现会下降，主要源于词汇和词分布的变化。本文研究了任务算术（Task Arithmetic）技术，该技术通过简单的数学操作（如加法或减法）组合在不同任务或领域上预训练的大型语言模型权重，以实现无需额外微调的检索模型适应。该方法能够将多种任务和领域知识整合到单一模型中，从而在不同检索场景下实现有效的零样本适配。在公开的科学、生物医学及多语言数据集上的大量实验表明，该方法使最先进的重排序性能在NDCG@10上提升高达18%，在P@10上提升15%。除了这些实证优势，我们的分析还揭示了任务算术作为零样本学习与模型适配实用策略的优势与局限性。代码已公开于 https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。