轻量级模型融合图文信息,提升检索列表重排序效果。
When Vision Meets Texts in Listwise Reranking
- 分步训练图文分支,用文本数据和大模型生成图像对来弥补数据不足
- 在多个图文检索基准上表现优异,最高达92.3%的MRR
- 仅用20亿参数,适合资源有限场景下的多模态应用
近年来,信息检索领域强调融合视觉与文本信息的潜力,但图文文档的重排序仍面临模态差异大、标注数据稀缺的挑战。现有方法多依赖70亿至320亿参数的大模型,通过基于推理的蒸馏方式,带来不必要的计算开销且主要关注文本模态。本文提出Rank-Nexus,一种基于列表的图文文档重排序器,利用图像与文本联合信息进行质量重排序。为弥合模态差距,我们采用渐进式跨模态训练策略:先分别训练文本与图像分支——文本分支利用丰富文本重排序数据进行知识蒸馏;图像分支则在图像检索基准上,通过多模态大语言模型(MLLM)生成的描述构建蒸馏样本对。随后,将两者结合蒸馏出联合图文重排序数据集。Rank-Nexus仅使用一个20亿参数的预训练视觉-语言模型,在文本重排序基准(TREC、BEIR)及具有挑战性的图文重排序基准(INQUIRE、MMDocIR)上均取得出色性能。该设计高效,具备强泛化能力,无需过多参数或推理开销即可适配多种多模态场景。
原文摘要 · Abstract (English)
Recent advancements in information retrieval have highlighted the potential of integrating visual and textual information, yet effective reranking for image-text documents remains challenging due to the modality gap and scarcity of aligned datasets. Meanwhile, existing approaches often rely on large models (7B to 32B parameters) with reasoning-based distillation, incurring unnecessary computational overhead while primarily focusing on textual modalities. In this paper, we propose Rank-Nexus, a multimodal image-text document reranker that performs listwise qualitative reranking on retrieved lists incorporating both images and texts. To bridge the modality gap, we introduce a progressive cross-modal training strategy. We first train modalities separately: leveraging abundant text reranking data, we distill knowledge into the text branch. For images, where data is scarce, we construct distilled pairs from multimodal large language model (MLLM) captions on image retrieval benchmarks. Subsequently, we distill a joint image-text reranking dataset. Rank-Nexus achieves outstanding performance on text reranking benchmarks (TREC, BEIR) and the challenging image reranking benchmark (INQUIRE, MMDocIR), using only a lightweight 2B pretrained visual-language model. This efficient design ensures strong generalization across diverse multimodal scenarios without excessive parameters or reasoning overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。