arXiv:2502.19712cs.IR2025-02被引 4

用大模型生成查询+列表蒸馏,让检索模型更懂专业领域。

Teaching Dense Retrieval Models to Specialize with Listwise Distillation and LLM Data Augmentation

  • 用交叉编码器的列表级相关性信号指导检索模型训练。
  • 在多个数据集上实现稳定提升,合成查询效果媲美人工查询。
  • 适合需要领域定制化检索的开发者与研究者。

当前先进的稠密检索模型虽具备良好的跨域泛化能力,但在捕捉细微领域知识方面表现不佳。理论上,针对特定任务微调应优于通用模型,但实际效果常不尽如人意。我们发现,使用InfoNCE损失的标准微调方法反而会降低性能,即使采用硬负样本挖掘和负样本去噪等常用技术也是如此。为此,我们提出一种基于教师交叉编码器的列表蒸馏策略,利用丰富的相关性信号来优化检索器。同时探索使用大语言模型生成合成查询,涵盖自然搜索、事实陈述和关键词查询等多种形式。通过这种组合策略,在多个数据集上均实现一致的有效性提升。结果表明,合成查询在训练中可媲美人工撰写查询。然而,我们也发现交叉编码器教师模型本身成为性能瓶颈。代码与脚本已开源,以促进后续研究。

原文摘要 · Abstract (English)

While the current state-of-the-art dense retrieval models exhibit strong out-of-domain generalization, they might fail to capture nuanced domain-specific knowledge. In principle, fine-tuning these models for specialized retrieval tasks should yield higher effectiveness than relying on a one-size-fits-all model, but in practice, results can disappoint. We show that standard fine-tuning methods using an InfoNCE loss can unexpectedly degrade effectiveness rather than improve it, even for domain-specific scenarios. This holds true even when applying widely adopted techniques such as hard-negative mining and negative de-noising. To address this, we explore a training strategy that uses listwise distillation from a teacher cross-encoder, leveraging rich relevance signals to fine-tune the retriever. We further explore synthetic query generation using large language models. Through listwise distillation and training with a diverse set of queries ranging from natural user searches and factual claims to keyword-based queries, we achieve consistent effectiveness gains across multiple datasets. Our results also reveal that synthetic queries can rival human-written queries in training utility. However, we also identify limitations, particularly in the effectiveness of cross-encoder teachers as a bottleneck. We release our code and scripts to encourage further research.

稠密检索模型蒸馏LLM生成领域适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。