arXiv:2508.04107cs.CVcs.AI2025-08被引 6

用轻量级解码器让多模态大模型精准分割图像区域,兼顾速度与精度。

Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder

  • 利用大模型视觉编码器的细节特征,不额外加视觉模块。
  • 设计融合细节与语义的模块,提升分割准确性。
  • 仅3400万参数的轻量解码器,适合资源受限场景使用。

指代表达分割(RES)旨在定位由自然语言描述指定的图像区域,随着多模态大模型(MLLMs)兴起而备受关注。尽管MLLM在语义理解上表现优异,但其基于标记生成的范式难以胜任像素级密集预测。现有方法或耦合参数庞大的Segment Anything Model(SAM,632M参数),或采用无SAM的轻量方案但牺牲精度。为此,本文提出MLLMSeg框架,充分挖掘MLLM视觉编码器中的内在视觉细节特征,无需引入额外视觉编码器。同时提出细节增强且语义一致的特征融合模块(DSFF),将视觉编码器输出的细节特征与大语言模型(LLM)生成的语义特征深度融合。最终构建仅含3400万参数的轻量级掩码解码器,有效利用视觉编码器的空间细节与LLM的语义信息,实现高精度掩码预测。大量实验表明,该方法普遍优于基于SAM及无SAM的基线,显著平衡了性能与计算成本。代码已开源:https://github.com/jcwang0602/MLLMSeg。

原文摘要 · Abstract (English)

Reference Expression Segmentation (RES) aims to segment image regions specified by referring expressions and has become popular with the rise of multimodal large models (MLLMs). While MLLMs excel in semantic understanding, their token-generation paradigm struggles with pixel-level dense prediction. Existing RES methods either couple MLLMs with the parameter-heavy Segment Anything Model (SAM) with 632M network parameters or adopt SAM-free lightweight pipelines that sacrifice accuracy. To address the trade-off between performance and cost, we specifically propose MLLMSeg, a novel framework that fully exploits the inherent visual detail features encoded in the MLLM vision encoder without introducing an extra visual encoder. Besides, we propose a detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully integrates the detail-related visual feature with the semantic-related feature output by the large language model (LLM) of MLLM. Finally, we establish a light-weight mask decoder with only 34M network parameters that optimally leverages detailed spatial features from the visual encoder and semantic features from the LLM to achieve precise mask prediction. Extensive experiments demonstrate that our method generally surpasses both SAM-based and SAM-free competitors, striking a better balance between performance and cost. Code is available at https://github.com/jcwang0602/MLLMSeg.

多模态图像分割轻量化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。