用动态路由让检索模型低成本适配新领域
Towards Dynamic Dense Retrieval with Routing Strategy
- 用前缀调优构建领域专用模块，再动态组合
- 仅用2%参数就超越传统方法，六项零样本任务表现更优
- 适合频繁更新模型的场景，节省训练成本
当前密集检索（DR）应用新任务的主流方法是微调预训练模型，但存在两大缺陷：一是当训练数据有限时难以适应新领域；二是旧模型被新模型替代时需从头训练，更新成本高昂。为解决这些问题，我们提出一种新型密集检索方法——动态密集检索（DDR）。DDR 使用前缀调优作为特定领域的模块，并通过动态路由策略实现模块组合，从而在检索部分实现高度灵活的领域适配。在六个零样本下游任务上的实证表明，该方法仅使用2%的训练参数即可超越传统密集检索，为信息检索中更灵活的模型应用开辟了新路径。
原文摘要 · Abstract (English)
The \textit{de facto} paradigm for applying dense retrieval (DR) to new tasks involves fine-tuning a pre-trained model for a specific task. However, this paradigm has two significant limitations: (1) It is difficult adapt the DR to a new domain if the training dataset is limited. (2) Old DR models are simply replaced by newer models that are trained from scratch when the former are no longer up to date. Especially for scenarios where the model needs to be updated frequently, this paradigm is prohibitively expensive. To address these challenges, we propose a novel dense retrieval approach, termed \textit{dynamic dense retrieval} (DDR). DDR uses \textit{prefix tuning} as a \textit{module} specialized for a specific domain. These modules can then be compositional combined with a dynamic routing strategy, enabling highly flexible domain adaptation in the retrieval part. Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR. We see it as a promising future direction for applying dense retrieval to various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。