让视觉模型自动决定何时检索，并用像素级图像查询提升问答准确率。
Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models
- 基于像素掩码生成视觉查询，直接触发检索，无需外部检测器
- 在CRAG-MM上比全图检索提升19.7%准确率，且保持强推理能力
- 适合需要精准视觉知识增强的多模态问答场景
视觉问答（VQA）常需结合细粒度感知与图像外的事实知识。现有多模态检索增强生成（MM-RAG）系统虽提升事实准确性，但缺乏内部检索时机与方式判断策略。本文提出PixSearch，首个端到端的分段式大视觉多模态模型（LMM），统一区域级感知与检索增强推理。编码阶段，PixSearch生成<search>标记以触发检索，自主选择查询模态（文本、图像或区域），并生成像素级掩码作为直接视觉查询，无需依赖模块化流水线（如检测器、分割器、描述器等）。通过两阶段监督微调，结合检索交错监督，学习检索时机与查询选择，同时保留分割能力。在自传式与实体中心型VQA基准上，PixSearch显著提升事实一致性与泛化能力，在CRAG-MM上相较全图检索实现19.7%相对准确率提升，且在各类VQA与纯文本问答任务中保持竞争力。
原文摘要 · Abstract (English)
Visual Question Answering (VQA) often requires coupling fine-grained perception with factual knowledge beyond the input image. Prior multimodal Retrieval-Augmented Generation (MM-RAG) systems improve factual grounding but lack an internal policy for when and how to retrieve. We propose PixSearch, the first end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning. During encoding, PixSearch emits <search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries, eliminating the reliance on modular pipelines (detectors, segmenters, captioners, etc.). A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval timing and query selection while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization, yielding a 19.7% relative gain in accuracy on CRAG-MM compared to whole image retrieval, while retaining competitive reasoning performance on various VQA and text-only QA tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。