用合成数据和列表式蒸馏提升检索模型效果，超越传统对比学习。
Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data
- 采用跨编码器列表蒸馏替代传统对比学习，提升模型一致性。
- 合成多类型查询数据，效果优于单一查询类型，且接近人工撰写质量。
- 在多个数据集上实现BERT模型最佳检索性能，适合信息检索研究者。
我们从语料库特定微调的角度研究如何提升嵌入模型的检索效果。先前工作表明，使用数据集检索语料库生成的查询进行微调可提升该数据集的检索效果。然而，我们发现令人意外的是，使用传统的InfoNCE对比损失微调反而会降低当前最先进模型的效果。为克服这一问题，我们重新审视跨编码器列表式蒸馏，发现相较于仅使用对比学习，列表蒸馏能在多个数据集上更一致地提升检索效果。此外，我们证明通过多种查询类型（如声明、关键词、问题）合成训练数据，比单独使用任一类型更具优势，且无论评估时使用何种查询类型，效果均更优。研究还表明，合成查询在训练中具有与人工撰写查询相当的价值。我们基于此方法训练出一个在BERT嵌入模型中达到最先进检索效果的模型，并公开了模型及查询生成与训练代码，以促进后续研究。
原文摘要 · Abstract (English)
We investigate improving the retrieval effectiveness of embedding models through the lens of corpus-specific fine-tuning. Prior work has shown that fine-tuning with queries generated using a dataset's retrieval corpus can boost retrieval effectiveness for the dataset. However, we find that surprisingly, fine-tuning using the conventional InfoNCE contrastive loss often reduces effectiveness in state-of-the-art models. To overcome this, we revisit cross-encoder listwise distillation and demonstrate that, unlike using contrastive learning alone, listwise distillation can help more consistently improve retrieval effectiveness across multiple datasets. Additionally, we show that synthesizing more training data using diverse query types (such as claims, keywords, and questions) yields greater effectiveness than using any single query type alone, regardless of the query type used in evaluation. Our findings further indicate that synthetic queries offer comparable utility to human-written queries for training. We use our approach to train an embedding model that achieves state-of-the-art effectiveness among BERT embedding models. We release our model and both query generation and training code to facilitate further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。