ReT-2统一支持图文混合查询与文档的跨模态检索，性能领先。
Recurrence Meets Transformers for Universal Multimodal Retrieval
- 采用循环式Transformer架构，动态融合多层图文特征。
- 在M2KR和M-BEIR上达到当前最优，推理更快、内存更少。
- 适合需要多模态检索增强生成任务的研究者使用。
随着多模态检索的发展及其在大语言模型和多模态大模型中的应用，越来越多复杂的检索任务涌现。现有方法主要依赖视觉-语言模型的任务特定微调，且仅限于单模态查询或文档。本文提出ReT-2，一种统一的检索模型，支持图文混合查询，并可在包含文本与图像的多模态文档集合中进行检索。ReT-2利用多层表示和受LSTM启发的门控机制的循环Transformer架构，动态整合跨层与跨模态信息，捕捉细粒度的视觉与文本细节。我们在M2KR和M-BEIR基准上评估了ReT-2在不同检索配置下的表现。结果表明，ReT-2在多种设置下持续取得最先进性能，同时相比以往方法具备更快的推理速度和更低的内存消耗。当集成至检索增强生成流水线时，ReT-2也在Encyclopedic-VQA和InfoSeek数据集上提升了下游任务表现。代码与训练模型已公开：https://github.com/aimagelab/ReT-2
原文摘要 · Abstract (English)
With the rapid advancement of multimodal retrieval and its application in LLMs and multimodal LLMs, increasingly complex retrieval tasks have emerged. Existing methods predominantly rely on task-specific fine-tuning of vision-language models and are limited to single-modality queries or documents. In this paper, we propose ReT-2, a unified retrieval model that supports multimodal queries, composed of both images and text, and searches across multimodal document collections where text and images coexist. ReT-2 leverages multi-layer representations and a recurrent Transformer architecture with LSTM-inspired gating mechanisms to dynamically integrate information across layers and modalities, capturing fine-grained visual and textual details. We evaluate ReT-2 on the challenging M2KR and M-BEIR benchmarks across different retrieval configurations. Results demonstrate that ReT-2 consistently achieves state-of-the-art performance across diverse settings, while offering faster inference and reduced memory usage compared to prior approaches. When integrated into retrieval-augmented generation pipelines, ReT-2 also improves downstream performance on Encyclopedic-VQA and InfoSeek datasets. Our source code and trained models are publicly available at: https://github.com/aimagelab/ReT-2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。