让大模型既能推理又能精准分割,靠视觉提示和多模态思维链。
RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought
- 用多模态思维链提示引导模型推理并定位目标区域。
- 在ReasonSeg上比顶尖方法高6.5 gIoU,零样本下SegInW达49.7 mAP。
- 适合需要可解释视觉推理的场景,如医疗影像分析、智能监控。
多模态大语言模型虽具强大推理能力,却缺乏显式的视觉定位与分割机制,导致认知推理与视觉感知脱节。为此,我们提出基于视觉提示的推理分割框架 RSVP,将多步多模态推理与视觉理解统一。该框架分两阶段:第一阶段利用多模态思维链视觉提示,使模型理解问题并推断目标,生成可解释的区域提议,增强视觉定位;第二阶段通过视觉-语言分割模块(VLSM)融合文本与视觉线索,精修提议,输出精确分割掩码。通过显式建模多模态推理与分割的交互,RSVP开创了一种可解释的推理分割新范式。其利用大模型内在定位能力,不仅推理对象,还生成结构化视觉表示。大量实验表明,RSVP在ReasonSeg上超越现有方法最高达+6.5 gIoU和+9.2 cIoU,零样本下于SegInW达到49.7 mAP,验证了其有效性与可扩展性。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable reasoning capability while lack explicit mechanisms for visual grounding and segmentation, creating a gap between cognitive reasoning and visual perception. To bridge this gap, we introduce Reasoning Segmentation via Visual Prompting (RSVP), a novel framework that unifies multi-step multimodal reasoning with grounded visual understanding. RSVP is a two-stage structuralized framework that integrates reasoning-driven localization with segmentation refinement. In the reasoning stage, RSVP employs multimodal chain-of-thought visual prompts to help MLLMs understand queries and infer targets, generating interpretable region proposals that enhance visual grounding. In segmentation stage, RSVP refines these proposals with a Vision-Language Segmentation Module (VLSM), seamlessly integrates textual and visual cues to produce precise segmentation masks. By explicitly modelling the interaction between multimodal reasoning and segmentation, RSVP introduces a new paradigm for interpretable reasoning segmentation. It exploits MLLMs' inherent localization capabilities, enabling the models to not only reason about objects but also generate structured visual representations. Our extensive experiments demonstrate that RSVP achieves state-of-the-art performance, surpasses state-of-the-art methods by up to +6.5 gIoU and +9.2 cIoU on ReasonSeg, and achieves 49.7 mAP on SegInW under zero-shot settings. These results validate RSVP as an effective and scalable framework for integrating cognitive reasoning with structured visual understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。