无需训练,让多模态大模型直接做检索,效果超越大量训练模型。
FreeRet: MLLMs as Training-Free Retrievers
- 利用预训练模型直接生成语义嵌入,跳过传统对齐层。
- 在46个数据集上表现优于需百万级训练对的模型。
- 适合想快速部署多模态检索且不想微调的研究者。
多模态大语言模型(MLLM)正成为混合模态检索的通用基础,但通常需要大量后处理训练才能转化为对比编码器。本文提出FreeRet,一种即插即用框架,将任意MLLM转为两阶段检索器:先用模型直接生成语义嵌入进行快速候选搜索,再利用其推理能力进行精确重排序。该框架有三大创新:跳过词汇对齐层以获得更忠实的语义嵌入;通过显式先验条件化表示生成;通过中性选择表述缓解重排序中的框架效应。在涵盖46个数据集的MMEB和MMEB-V2基准上,FreeRet显著优于需数百万训练对的模型。此外,FreeRet具有模型无关性,可无缝扩展至不同规模的MLLM家族,保留生成能力,支持任意模态组合,并在单一模型内统一检索、重排序与生成,实现端到端RAG。结果表明,经过精心设计,预训练MLLM无需训练即可作为强大检索引擎,填补其作为通用模型的关键空白。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) are emerging as versatile foundations for mixed-modality retrieval. Yet, they often require heavy post-hoc training to convert them into contrastive encoders for retrieval. This work asks: Can off-the-shelf MLLMs serve as powerful retrievers without additional training? We present FreeRet, a plug-and-play framework that turns any MLLM into a two-stage retriever. FreeRet first derives semantically grounded embeddings directly from the model for fast candidate search, and then exploits its reasoning ability for precise reranking. The framework contributes three advances: bypassing lexical alignment layers to obtain semantically faithful embeddings, conditioning representation generation with explicit priors, and mitigating framing effect in reranking via neutral choice framing. On the MMEB and MMEB-V2 benchmarks spanning 46 datasets, FreeRet substantially outperforms models trained on millions of pairs. Beyond benchmarks, FreeRet is model-agnostic and scales seamlessly across MLLM families and sizes, preserves their generative abilities, supports arbitrary modality combinations, and unifies retrieval, reranking, and generation into end-to-end RAG within a single model. Our findings demonstrate that pretrained MLLMs, when carefully harnessed, can serve as strong retrieval engines without training, closing a critical gap in their role as generalists.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。