arXiv:2412.11832cs.IR2024-12

构建分布式协作检索框架,让所有模型在各类查询中表现更优

A Distributed Collaborative Retrieval Framework Excelling in All Queries and Corpora based on Zero-shot Rank-Oriented Automatic Evaluation

  • 融合多种检索模型,动态选择最优结果
  • 8个模型组合性能媲美RankGPT等方法,效率更高
  • 无需标注数据,用LLM提示词自动评估排序质量

众多稀疏、密集及基于大语言模型的检索方法在预测查询与文档相关性方面表现出色。然而初步有效性分析显示,这些模型在多数查询和语料上表现不佳,其效果受限于特定场景。为此,本文提出一种新型分布式协作检索框架(DCRF),在所有查询和语料上均优于单一模型。该框架将多种检索模型集成到统一系统中,动态为每个用户查询选择最优结果,可灵活整合任意检索模型并扩展至各类应用场景。此外,为降低维护与训练成本,设计了四种有效的提示策略,利用大语言模型实现无需标注数据的排名质量自动评估。大量实验表明,结合8个高效检索模型的DCRF,在性能上可媲美如RankGPT和ListT5等先进的列表式方法,同时具备更高效率。且在多数数据集上超越所有选定模型,验证了提示策略在排名导向自动评估中的有效性。

原文摘要 · Abstract (English)

Numerous retrieval models, including sparse, dense and llm-based methods, have demonstrated remarkable performance in predicting the relevance between queries and corpora. However, the preliminary effectiveness analysis experiments indicate that these models fail to achieve satisfactory performance on the majority of queries and corpora, revealing their effectiveness restricted to specific scenarios. Thus, to tackle this problem, we propose a novel Distributed Collaborative Retrieval Framework (DCRF), outperforming each single model across all queries and corpora. Specifically, the framework integrates various retrieval models into a unified system and dynamically selects the optimal results for each user's query. It can easily aggregate any retrieval model and expand to any application scenarios, illustrating its flexibility and scalability.Moreover, to reduce maintenance and training costs, we design four effective prompting strategies with large language models (LLMs) to evaluate the quality of ranks without reliance of labeled data. Extensive experiments demonstrate that proposed framework, combined with 8 efficient retrieval models, can achieve performance comparable to effective listwise methods like RankGPT and ListT5, while offering superior efficiency. Besides, DCRF surpasses all selected retrieval models on the most datasets, indicating the effectiveness of our prompting strategies on rank-oriented automatic evaluation.

检索框架大模型自动评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。