arXiv:2503.23239cs.IRcs.CL2025-03EMNLP被引 4

用合成数据生成多级相关性,提升检索模型排序效果。

Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance

  • 用大模型生成不同相关程度的合成文档,替代传统正负样本
  • 在MS MARCO和BEIR上超越InfoNCE方法,提升显著
  • 适合追求高精度排序与自监督训练的检索研究者

尽管合成数据已改变信息检索多个环节,主流训练范式仍为基于二元相关标签的对比学习,使用InfoNCE损失将一个正例与多个负例对比。该目标将所有未明确标注为相关的文档视为同等负例,忽略了实际相关性的细微差异。为此,本文摒弃真实文档与标注,利用大语言模型直接生成符合多级别相关性的合成文档以回应MS MARCO查询。同时提出使用Wasserstein距离作为更有效的损失函数,训练具有渐进相关标签的Transformer检索器。在MS MARCO和BEIR基准上的实验表明,所提方法相比传统InfoNCE训练大幅领先。不使用任何真实文档,该方法显著提升自监督检索器性能,并对分布偏移更具鲁棒性。此外,该方法可有效融合现有真实数据,进一步提升性能。总体而言,生成多级排序上下文是信息检索中合成数据生成更优方案。

原文摘要 · Abstract (English)

Although synthetic data has changed various aspects of information retrieval (IR) pipelines, the main training paradigm remains: contrastive learning with binary relevance labels, where one positive document is compared against several negatives using the InfoNCE loss. This objective treats all documents that are not explicitly annotated as relevant on an equally negative footing, regardless of their actual degree of relevance, thus missing subtle nuances useful for ranking. To overcome this limitation, in this work, we forgo real documents and annotations and use large language models to directly generate synthetic documents that answer the MS MARCO queries according to several different levels of relevance. We also propose using Wasserstein distance as a more effective loss function for training transformer-based retrievers with graduated relevance labels. Our experiments on MS MARCO and BEIR benchmark show that our proposed approach outperforms conventional training with InfoNCE by a large margin. Without using any real documents, our method significantly improves self-supervised retrievers and is more robust to distribution shift compared to contrastive learning using real data. Our method also successfully integrates existing real data into the synthetic ranking context, further boosting the performance. Overall, we show that generating multi-level ranking contexts is a better approach to synthetic data generation for IR than just generating the standard positive and negative documents.

信息检索合成数据多级相关Wasserstein

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。