arXiv:2509.06321cs.CV2025-09TPAMI被引 9

用文本生成方式做图像分割,无需额外解码器。

Text4Seg++: Advancing Image Segmentation via Generative Language Modeling

  • 把分割结果转成文本描述,让大模型直接生成
  • 压缩文本长度74%,推理速度提升3倍,性能不降
  • 适合想用通用大模型做分割的开发者

多模态大语言模型在视觉-语言任务中表现优异,但如何有效融入图像分割仍具挑战。本文提出一种新的「文本即掩码」范式,将图像分割转化为文本生成问题,省去额外解码器,显著简化流程。核心创新是语义描述符——将每个图像块映射为对应文本标签的新型文本表示。我们首先引入图像级语义描述符,实现与语言建模流水线的自然融合;为提升效率,提出行级运行长度编码(R-RLE),使语义描述符长度减少74%,推理速度提升3倍,且不影响性能。基于此,初始框架Text4Seg在多种视觉任务中取得强分割效果。为进一步提升精度与紧凑性,我们提出框级语义描述符,利用边界框定位感兴趣区域,并通过结构化掩码标记(语义积木)表示区域掩码。由此构建的Text4Seg++将分割定义为“预测下一个积木”任务,兼顾精度、可扩展性与生成效率。在自然图像与遥感数据集上的全面实验表明,Text4Seg++在无需任务微调的情况下,持续优于现有最先进模型,且兼容现有MLLM骨干网络。本工作凸显了文本驱动分割在MLLM框架下的有效性、可扩展性与通用性。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks. However, effectively integrating image segmentation into these models remains a significant challenge. In this work, we propose a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. We first introduce image-wise semantic descriptors, a patch-aligned textual representation of segmentation masks that integrates naturally into the language modeling pipeline. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74% and accelerating inference by $3\times$, without compromising performance. Building upon this, our initial framework Text4Seg achieves strong segmentation performance across a wide range of vision tasks. To further improve granularity and compactness, we propose box-wise semantic descriptors, which localizes regions of interest using bounding boxes and represents region masks via structured mask tokens called semantic bricks. This leads to our refined model, Text4Seg++, which formulates segmentation as a next-brick prediction task, combining precision, scalability, and generative efficiency. Comprehensive experiments on natural and remote sensing datasets show that Text4Seg++ consistently outperforms state-of-the-art models across diverse benchmarks without any task-specific fine-tuning, while remaining compatible with existing MLLM backbones. Our work highlights the effectiveness, scalability, and generalizability of text-driven image segmentation within the MLLM framework.

图像分割文本生成大模型语义描述

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。