arXiv:2410.09855cs.CV2024-10ICLR被引 65

把图像分割变成文本生成,用文字描述代替传统掩码。

Text4Seg: Reimagining Image Segmentation as Text Generation

  • 将图像块映射为文本标签,构建统一的文本化分割表示。
  • 16×16语义描述符实现媲美主流方法的分割性能。
  • 采用行级游程编码压缩文本,推理速度提升3倍且不损失精度。

多模态大语言模型在视觉-语言任务中表现卓越,但如何有效融入图像分割仍具挑战。本文提出Text4Seg,一种将图像分割重构为文本生成的新范式,无需额外解码器,显著简化流程。核心创新是引入语义描述符——将每个图像块映射为其对应文本标签,形成统一文本表示,可直接嵌入MLLM的自回归训练框架以实现高效优化。实验表明,使用$16\times16$语义描述符即可获得具有竞争力的分割效果。为提升效率,提出行级游程编码(R-RLE),将冗余文本序列压缩74%,使语义描述符长度减少,推理速度提升3倍,且性能无损。在指代表达分割与理解等多类视觉任务上,通过微调不同MLLM主干,Text4Seg在多个数据集上达到领先性能。该方法为MLLM框架下的视觉任务提供了高效可扩展的解决方案。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks; however, effectively integrating image segmentation into these models remains a significant challenge. In this paper, we introduce Text4Seg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. This unified representation allows seamless integration into the auto-regressive training pipeline of MLLMs for easier optimization. We demonstrate that representing an image with $16\times16$ semantic descriptors yields competitive segmentation performance. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74% and accelerating inference by $3\times$, without compromising performance. Extensive experiments across various vision tasks, such as referring expression segmentation and comprehension, show that Text4Seg achieves state-of-the-art performance on multiple datasets by fine-tuning different MLLM backbones. Our approach provides an efficient, scalable solution for vision-centric tasks within the MLLM framework.

图像分割文本生成多模态模型语义描述

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。