无需训练和数据，用提示工程让大模型直接做多模态检索
RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
- 用粗筛+细调两阶段提示，让大模型直接预测检索得分
- 在多个基准上超越微调模型，零训练仍达顶尖性能
- 适合想快速部署多模态检索的开发者和研究者
多模态信息检索（MMIR）因其可处理文本、图像或混合查询与候选内容而受到关注。近期基于多模态大语言模型（MLLM）的突破，在对比微调框架下融入了MLLM知识，显著提升了MMIR性能。然而，现有方法存在预训练不一致问题，且依赖大规模数据集。本文提出一种新框架RetLLM，实现无训练、无数据的多模态信息检索。我们把MMIR建模为相似度得分生成任务，采用粗筛-细调提示管道，先通过top-k筛选构建高质量候选池，再由MLLM联合输入查询与候选预测得分。关键创新在于推理阶段引入视觉增强模块，帮助模型重新关注被忽略的视觉信息。在多个MMIR基准上的实验表明，RetLLM优于微调模型。消融实验证明各组件有效性。结果表明，仅通过提示工程，大模型即可实现强大检索能力，凸显其固有的多模态推理潜力。代码已开源：https://github.com/alivecat05/RETLLM
原文摘要 · Abstract (English)
Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。