通过动态裁剪关键区域，提升多模态重排序的准确性
Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking
- 将区域选择建模为决策问题，动态裁剪问题相关区域
- 在两个基准上最高提升条件召回率20%
- 适合需要精准图文匹配的检索增强生成场景
多模态检索增强生成（MM-RAG）严重依赖重排序器来提取与图像-问题查询最相关的证据。然而，标准重排序器通常以全局嵌入方式处理完整查询图像，容易受视觉干扰（如背景杂乱）影响，导致相似度评分偏差。我们提出Region-R1，一种查询端区域裁剪框架，将区域选择建模为重排序过程中的决策问题，使系统学会保留全图或仅聚焦于与问题相关的区域后再对候选内容进行打分。Region-R1通过一种新颖的区域感知组相对策略优化（r-GRPO）学习裁剪策略，动态选取判别性强的区域。在两个挑战性基准E-VQA和InfoSeek上，Region-R1均取得一致性能提升，达到当前最优表现，条件召回率@1最高提升20%。结果表明，查询端自适应是一种简单但有效的强化MM-RAG重排序的方法。
原文摘要 · Abstract (English)
Multi-modal retrieval-augmented generation (MM-RAG) relies heavily on re-rankers to surface the most relevant evidence for image-question queries. However, standard re-rankers typically process the full query image as a global embedding, making them susceptible to visual distractors (e.g., background clutter) that skew similarity scores. We propose Region-R1, a query-side region cropping framework that formulates region selection as a decision-making problem during re-ranking, allowing the system to learn to retain the full image or focus only on a question-relevant region before scoring the retrieved candidates. Region-R1 learns a policy with a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region. Across two challenging benchmarks, E-VQA and InfoSeek, Region-R1 delivers consistent gains, achieving state-of-the-art performances by increasing conditional Recall@1 by up to 20%. These results show the great promise of query-side adaptation as a simple but effective way to strengthen MM-RAG re-ranking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。