让稀疏与密集检索互相学习,提升图文检索效果
Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval
- 用双向知识蒸馏共享相似度信号,联合优化稀疏与密集表示
- 在MSCOCO和Flickr30k上,稀疏检索器性能媲美甚至超过密集模型
- 只需微调末层,适配现有视觉语言模型,高效且易部署
视觉语言预训练(VLP)模型在多模态任务中表现优异,尤其依赖密集表示进行图文检索。与此同时,学习型稀疏检索(LSR)因其可解释性与高效性,在纯文本场景中广受关注,可通过倒排索引实现快速词项查找。受此启发,近期工作已将LSR拓展至多模态领域。然而,这些方法常依赖计算开销大的对比预训练或从冻结的密集模型中进行蒸馏,限制了二者间的相互增强。为此,本文提出一种简单而有效的框架,通过自知识蒸馏实现稠密与稀疏表示之间的双向学习。双向学习基于一个综合相似度分数——稠密与稀疏相似度的加权和——作为两者的共同教师信号。为保证效率,仅微调稠密编码器的最后层和稀疏投影头,使任何现有VLP模型均可轻松适配。在MSCOCO和Flickr30k上的实验表明,所提稀疏检索器不仅优于现有稀疏基线,其性能还可媲美甚至超越对应稠密模型,同时保留稀疏模型的高效优势。
原文摘要 · Abstract (English)
Vision-Language Pretrained (VLP) models have achieved impressive performance on multimodal tasks, including text-image retrieval, based on dense representations. Meanwhile, Learned Sparse Retrieval (LSR) has gained traction in text-only settings due to its interpretability and efficiency with fast term-based lookup via inverted indexes. Inspired by these advantages, recent work has extended LSR to the multimodal domain. However, these methods often rely on computationally expensive contrastive pre-training, or distillation from a frozen dense model, which limits the potential for mutual enhancement. To address these limitations, we propose a simple yet effective framework that enables bi-directional learning between dense and sparse representations through Self-Knowledge Distillation. This bi-directional learning is achieved using an integrated similarity score-a weighted sum of dense and sparse similarities-which serves as a shared teacher signal for both representations. To ensure efficiency, we fine-tune the final layer of the dense encoder and the sparse projection head, enabling easy adaptation of any existing VLP model. Experiments on MSCOCO and Flickr30k demonstrate that our sparse retriever not only outperforms existing sparse baselines, but also achieves performance comparable to-or even surpassing-its dense counterparts, while retaining the benefits of sparse models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。