arXiv:2502.07555cs.CL2025-02被引 12

让检索模型先思考再行动,提升复杂查询的精准度。

O1 Embedder: Let Retrievers Think Before Action

  • 引入思考阶段,先生成查询意图再进行文档检索。
  • 在12个数据集上均显著优于现有方法,跨域表现优异。
  • 适合需要深度推理的检索任务,如多任务、零样本场景。

大型语言模型(LLM)的兴起彻底改变了信息获取方式,其在细粒度数据表征和基于外部参考生成高质量答案方面表现出色。近期推理模型(如OpenAI O1、DeepSeek R1)的出现,进一步展现了模型在输出最终答案前逐步思考的能力,显著提升了处理复杂任务(如编程、数学证明)的效果。受此启发,我们致力于为检索模型赋予类似能力,以应对多任务检索、零样本检索及复杂关系推理等关键挑战。为此提出O1 Embedder,该模型在执行检索前,先为输入查询生成有用的思想。为实现这一目标,我们攻克两大技术难点:一是设计数据合成流程,通过LLM专家生成初始思想,并由检索委员会进行精炼;二是优化训练过程,使预训练模型通过行为克隆生成检索思想,并通过对比学习完成密集检索。在12个主流数据集上的全面实验表明,该方法在域内与域外场景中均有显著提升,充分体现了O1 Embedder在准确性与泛化性方面的卓越表现,为下一代信息检索基础模型的发展铺平道路。

原文摘要 · Abstract (English)

The growing power of large language models (LLMs) has revolutionized how people access and utilize information. Notably, the LLMs excel at performing fine-grained data representation, which facilitates precise retrieval of information. They also generate high-quality answers based on external references, enabling the production of useful knowledge. The recent introduction of reasoning models, like OpenAI O1 and DeepSeek R1, marks another leap forward, highlighting LLMs' ability to think progressively before delivering final answers. This breakthrough significantly improves the ability to address complex tasks, e.g., coding and math proofs. Inspired by this progress, we aim to develop similar capabilities for retrieval models, which hold great promise for tackling critical challenges in the field, including multi-task retrieval, zero-shot retrieval, and tasks requiring intensive reasoning of complex relationships. With this motivation, we propose a novel approach called O1 Embedder, which generates useful thoughts for the input query before making retrieval for the target documents. To realize this objective, we conquer two technical difficulties. First, we design a data synthesis workflow, creating training signals for O1 Embedder by generating initial thoughts from an LLM-expert and subsequently refining them using a retrieval committee. Second, we optimize the training process, enabling a pre-trained model to be jointly fine-tuned to generate retrieval thoughts via behavior cloning and perform dense retrieval through contrastive learning. Our approach is evaluated by comprehensive experiments, where substantial improvements are achieved across 12 popular datasets, spanning both in-domain and out-of-domain scenarios. These results highlight O1 Embedder's remarkable accuracy and generalizability, paving the way for the development of next-generation IR foundation models.

信息检索思考机制大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。