用路由机制选最适合的专家模型,提升跨领域检索效果。
RouterRetriever: Routing over a Mixture of Expert Embedding Models
- 根据查询动态选择最匹配的领域专家嵌入模型
- 在BEIR上比通用模型高2.1点nDCG@10,比多任务模型高3.2点
- 轻量设计,可灵活增删专家,适合多领域检索场景
信息检索通常依赖于在大规模通用数据集(如MSMARCO)上训练的单一嵌入模型。尽管整体表现尚可,但在特定领域测试时性能常不及领域专用模型。以往工作尝试通过多任务训练解决,但尚未探索在检索中对领域专家模型进行路由选择。本文提出RouterRetriever,利用路由机制为每个查询选择最合适的领域专家嵌入模型。该模型轻量且可灵活增删专家,无需重新训练。在BEIR基准上的评估显示,其性能优于MSMARCO训练模型(+2.1绝对nDCG@10)和多任务模型(+3.2)。路由机制平均优于语言模型常用方法(+1.8)。该优势在无专属专家的数据集上也有效。RouterRetriever是首个证明在多样化专业领域中,通过路由专家模型替代通用模型的有效性的工作。
原文摘要 · Abstract (English)
Information retrieval methods often rely on a single embedding model trained on large, general-domain datasets like MSMARCO. While this approach can produce a retriever with reasonable overall performance, they often underperform models trained on domain-specific data when testing on their respective domains. Prior work in information retrieval has tackled this through multi-task training, but the idea of routing over a mixture of domain-specific expert retrievers remains unexplored despite the popularity of such ideas in language model generation research. In this work, we introduce RouterRetriever, a retrieval model that leverages a mixture of domain-specific experts by using a routing mechanism to select the most appropriate expert for each query. RouterRetriever is lightweight and allows easy addition or removal of experts without additional training. Evaluation on the BEIR benchmark demonstrates that RouterRetriever outperforms both models trained on MSMARCO (+2.1 absolute nDCG@10) and multi-task models (+3.2). This is achieved by employing our routing mechanism, which surpasses other routing techniques (+1.8 on average) commonly used in language modeling. Furthermore, the benefit generalizes well to other datasets, even in the absence of a specific expert on the dataset. RouterRetriever is the first work to demonstrate the advantages of routing over a mixture of domain-specific expert embedding models as an alternative to a single, general-purpose embedding model, especially when retrieving from diverse, specialized domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。