解决多粒度查询中模型忽略细节信息的问题,提升跨模态检索精度。
ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

- 基于规则奖励的强化学习框架,通过排序驱动优化负样本区分度。
- 在标准基准上达到顶尖性能,MRBench上提升13.1%显著缓解粒度盲区。
- 适用于复杂多粒度查询场景,适合关注检索细粒度语义的研究者。
利用对比学习增强多模态大语言模型(MLLM)已成为提升通用多模态检索(UMR)性能的主流方法。然而,现有方法忽略了将对比范式应用于检索任务时存在的粒度盲区问题——即模型容易忽视查询中蕴含的细粒度信息,而这对于处理复杂查询至关重要。根源在于对比学习将样本视为二分类(正/负),忽略了每个负样本携带的信息差异。为此,我们提出一种简单有效的框架ELVA,通过排序驱动的MLLM缓解粒度盲区。1)不依赖奖励模型,将可验证奖励强化学习(RLVR)拓展至检索任务,使模型在无显式排序标签下探索新排序行为;2)采用规则奖励机制,联合优化负样本排序并扩大正负样本间相似度差距。为更精准评估粒度盲区,我们构建了专门针对多粒度查询场景的MRBench基准。ELVA在标准检索基准上取得领先结果,且在MRBench上实现13.1%的显著提升,充分验证其有效性。
原文摘要 · Abstract (English)
Leveraging Multimodal Large Language Models (MLLMs) via contrastive learning has become a mainstream paradigm for improving the performance of Universal Multimodal Retrieval (UMR). However, previous works have ignored the grain blindness when adapting the contrastive paradigm into retrieval tasks. Grain blindness refers to the tendency of the model to overlook grain-level information contained in the query, which is crucial for effectively handling complex queries. This stems from contrastive learning treating samples as a binary classification (positive/negative), while ignoring the different information carried by each negative sample. To address this, we argue that negatives should be treated differently according to their similarity to the positive sample, enabling the model to learn distinct grain information from each negative. In this paper, we introduce a simple but effective framework, called ELVA, a novel rule-based RL framework that mitigates grain blindness through ranking-driven MLLMs. 1) Instead of relying on reward models, we extend Reinforcement Learning with Verifiable Rewards (RLVR) to retrieval tasks, allowing the model to explore new ranking behaviors without explicit ranking labels. 2) By utilizing rule-based rewards, our approach jointly optimizes the ranking of negative samples while enlarging the similarity gap between positive and negative. To more precisely measure grain blindness, we further introduce MRBench, a new benchmark specifically designed for multi-grain query scenarios. ELVA achieves state-of-the-art results across standard retrieval benchmarks, and its notable 13.1% improvement on MRBench further demonstrates its effectiveness in alleviating grain blindness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。