用语言模型精准定位图像分割点，无需改架构也不增计算量。
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation
- 通过提问方式让语言模型找出发射点，驱动SAM进行像素级分割。
- 在公开数据集上显著提升分割精度，且不增加额外训练开销。
- 适合想低成本接入高精度图像分割的视觉语言任务研究者。
我们提出SAM4MLLM，一种将分割一切模型（SAM）与多模态大语言模型（MLLM）融合的新方法，用于像素感知任务。该方法使MLLM在不修改现有模型结构或添加专用标记的情况下，学习像素级位置信息。通过基于询问的策略，有效为SAM生成分割所需的提示点，以统一的语言形式结合详尽的视觉信息与大语言模型的强大表达能力，且学习过程无额外计算开销。在多个公开基准上的实验结果证明了该方法的有效性。
原文摘要 · Abstract (English)
We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models (MLLMs) for pixel-aware tasks. Our method enables MLLMs to learn pixel-level location information without requiring excessive modifications to the existing model architecture or adding specialized tokens. We introduce an inquiry-based approach that can effectively find prompt points for SAM to perform segmentation based on MLLM. It combines detailed visual information with the powerful expressive capabilities of large language models in a unified language-based manner without additional computational overhead in learning. Experimental results on pubic benchmarks demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。