arXiv:2601.16093cs.CV2026-01被引 8

用两个词表示任意图像掩码,让大模型轻松实现像素级理解。

SAMTok: Representing Any Mask with Two Words

  • 将掩码转为两个离散词元,用标准语言模型训练即可
  • 在2090万掩码上训练,500万数据微调后性能领先
  • 无需修改模型结构,适合想加像素能力的开发者

像素级能力是构建交互式智能系统的关键,但现有的多模态大模型因区域编码器复杂、分割解码器专用及训练目标不兼容,难以扩展。为此,我们提出SAMTok,一种将任意区域掩码转换为两个特殊词元的离散掩码分词器,并能高保真重建掩码。通过将掩码视为新语言词元,SAMTok使基础多模态大模型(如QwenVL系列)仅通过标准下一词预测和简单强化学习即可获得像素级能力,无需架构修改或专门损失设计。SAMTok基于SAM2,在2090万多样化掩码上训练,使用掩码编码器与残差向量量化生成离散、紧凑且信息丰富的词元。结合500万经SAMTok格式化的掩码理解与生成数据,QwenVL-SAMTok在区域描述、区域VQA、有根基对话、指代分割、场景图解析及多轮交互分割任务中达到或超过当前最优表现。我们进一步引入文本答案匹配奖励,实现高效强化学习,显著提升GRES和GCG基准表现。结果表明,该方法为赋予多模态大模型强大像素级能力提供了一种可扩展且简便的新范式。代码与模型已开源。

原文摘要 · Abstract (English)

Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask using these tokens with high fidelity. By treating masks as new language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.

多模态掩码表示大模型像素级

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。