提出结构化全掩码预测，实现快速高精度多目标分割且不损失对话能力。
Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
- 分离对话与掩码预测，单步完成所有目标的非自回归生成
- 支持开放词汇语义、实例感知和遥感小目标分割，精度领先
- 推理速度提升62%，保留模型多模态理解能力，适合复杂视觉任务
基于多模态大模型的分割面临核心三难困境：高分割性能、保持对话能力与快速推理。嵌入-预测方法可能破坏语言建模，而下一个词生成对密集掩码效率低下。我们提出全掩码预测，将自回归对话与非自回归掩码预测解耦。其二值版本STAMP（Simultaneous Textual All-Mask Prediction）发出词表内<SEG>触发符，融合图像对齐掩码标记与对应图像块特征，并使用混合注意力在一步中分类所有标记为前景或背景。该方法结合强指代与推理分割能力、保持多模态能力并实现高效推理。然而二值掩码无法保留多个语义或实例身份，需重复目标特定预测。因此我们提出结构化全掩码预测并开发STAMPlus。它生成带显式ID和可选框的目标列表，将这些ID绑定到共享多类掩码空间，联合预测所有目标于一次非自回归传递中。单一统一检查点保留STAMP的指代与推理能力，扩展至开放词汇语义、实例感知及遥感小目标分割，其中高分辨率掩码标记缩放保留更细空间证据。在这些设置下，STAMPlus达到最先进分割性能，保持通用多模态指令遵循能力，12类任务延迟从重复STAMP推理的13.50秒降至5.16秒。进一步分析表明，准确目标提示提升分割效果，学习到的空间定位有助于看两次推理。总体而言，STAMPlus超越单目标预测，解决了三难困境。
原文摘要 · Abstract (English)
MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction. Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary <SEG> trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference. However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass. A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s. Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。