用一个分割标记让多模态大模型直接完成图像分割,无需额外解码器。
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
- 仅用一个分割嵌入,通过特征重构提升分辨率。
- 在多个数据集上达到与专用解码器相当的分割精度。
- 适合想简化分割流程、追求轻量化部署的研究者。
近期基于多模态大语言模型(MLLM)的分割方法展现出可靠的物体级分割能力与空间感知。然而,几乎所有方法都依赖专用掩码解码器来解析生成的分割嵌入或引入多个附加标记辅助。本文探索是否能仅凭一个分割嵌入(SELF1E)从MLLM自身实现分割,且无需外部解码器。针对MLLM中像素打乱导致的分辨率下降问题,我们保留原始未压缩图像特征,并用MLLM处理后的压缩特征提取残差特征进行填充,提升特征精度。随后,分别对经过和未经过LLM处理的图像特征执行反像素打乱操作,释放压缩特征细节并放大未压缩分辨率下的残差特征,进一步增强重构质量。此外,设计双感知路径注意力掩码,实现像素与分割标记间的丰富特征交互。在多个分割任务上的实验证明,SELF1E性能可媲美依赖专用解码器的方法,证实了无解码器分割在MLLM中的可行性。项目页:https://github.com/ANDYZAQ/SELF1E。
原文摘要 · Abstract (English)
Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。