让AI看图问答更准：先规划要找什么图，再精准筛选并重排。
R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation
- 先生成视觉线索需求计划，再分两步找图：粗筛后精排。
- 在六个大模型上提升准确率，九个场景全胜，达当前最佳。
- 适合需要高精度图文理解的场景，如医疗、科研图像分析。
面向视觉中心的视觉问答任务，需检索图像以补全缺失的视觉线索，并将其融入推理过程。然而，如何选择合适图像并有效整合至模型推理仍具挑战。为此，我们提出R3G——一种模块化推理-检索-重排框架。该框架首先生成简明的推理计划，明确所需视觉线索；随后采用两阶段策略，先粗粒度检索，再细粒度重排，筛选证据图像。在MRAG-Bench基准上，R3G在六种多模态大模型（MLLM）和九个子场景中均实现准确率提升，达到当前最优整体性能。消融实验表明，充分性感知的重排与推理步骤具有互补性，使模型既能选对图像，也能用好图像。代码与数据已开源于https://github.com/czh24/R3G。
原文摘要 · Abstract (English)
Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process. However, selecting the right images and integrating them effectively into the model's reasoning remains challenging. To address this challenge, we propose R3G, a modular Reasoning-Retrieval-Reranking framework. It first produces a brief reasoning plan that specifies the required visual cues, then adopts a two-stage strategy, with coarse retrieval followed by fine-grained reranking, to select evidence images. On MRAG-Bench, R3G improves accuracy across six MLLM backbones and nine sub-scenarios, achieving state-of-the-art overall performance. Ablations show that sufficiency-aware reranking and reasoning steps are complementary, helping the model both choose the right images and use them well. We release code and data at https://github.com/czh24/R3G.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。