提出新方法让大模型同时高效对话、精准分割、快速推理。
Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction
- 将分割任务转为并行填空,避免与对话目标冲突。
- 单次前向传播完成完整分割,速度远超传统方法。
- 在多个数据集上兼顾对话能力、分割精度和推理速度。
将分割集成到多模态大语言模型(MLLM)中面临核心三难困境:同时保持对话能力、实现高分割性能和确保快速推理。现有方法被迫妥协:嵌入预测引入像素级目标,损害对话通用性;下一词预测将分割重构为自回归任务,导致稀疏输出时性能差或丰富输出时推理速度极慢。本文提出全新范式——全掩码预测,解耦自回归对话生成与非自回归掩码预测。我们构建了STAMP:Simultaneous Textual All-Mask Prediction,该模型在生成文本回复后,通过将掩码预测视为图像块的并行“填空”任务,在单次前向传播中完成完整分割。此设计既避免了冲突目标,保留对话能力;又利用丰富的双向空间上下文提升分割精度;还实现卓越速度。大量实验表明,STAMP在多个分割基准上显著优于当前最优方法,提供了一种在对话、分割与速度上均无妥协的解决方案。
原文摘要 · Abstract (English)
Integrating segmentation into Multimodal Large Language Models (MLLMs) presents a core trilemma: simultaneously preserving dialogue ability, achieving high segmentation performance, and ensuring fast inference. Prevailing paradigms are forced into a compromise. Embedding prediction methods introduce a conflicting pixel-level objective that degrades the MLLM's general dialogue abilities. The alternative, next-token prediction, reframes segmentation as an autoregressive task, which preserves dialogue but forces a trade-off between poor segmentation performance with sparse outputs or prohibitive inference speeds with rich ones. We resolve this trilemma with all-mask prediction, a novel paradigm that decouples autoregressive dialogue generation from non-autoregressive mask prediction. We present STAMP: Simultaneous Textual All-Mask Prediction, an MLLM that embodies this paradigm. After generating a textual response, STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches. This design maintains the MLLM's dialogue ability by avoiding conflicting objectives, enables high segmentation performance by leveraging rich, bidirectional spatial context for all mask tokens, and achieves exceptional speed. Extensive experiments show that STAMP significantly outperforms state-of-the-art methods across multiple segmentation benchmarks, providing a solution that excels in dialogue, segmentation, and speed without compromise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。