让冻结的多模态大模型轻松实现像素级分割，无需重训练。
Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
- 通过轻量可训练头提取注意力图中的关键点，直接生成适配掩码解码器的点特征。
- 在多个数据集上性能媲美甚至超越微调方法，且保持模型原有泛化能力。
- 适合希望扩展多模态大模型能力又不破坏其通用性的研究者和开发者。
将多样视觉能力整合进统一模型是多模态大语言模型（MLLMs）的重要趋势。其中，分割任务面临独特挑战。现有方法需对模型进行微调以生成适配掩码解码器的输出，通常会改变模型输出空间并损害其内在泛化能力，违背统一模型目标。本文提出 LENS（Leveraging kEypoiNts for MLLMs' Segmentation），一种全新的即插即用方案。LENS 将一个轻量、可训练的头部附加到完全冻结的 MLLM 上，通过优化注意力图中的空间线索，提取关键点并转化为与掩码解码器直接兼容的点特征。大量实验验证：LENS 在分割性能上达到或优于微调方法，同时完整保留 MLLM 的泛化能力，而后者在微调中显著退化。因此，LENS 的可附加设计建立了一种高效且强大的扩展范式，为真正多才多艺的统一模型铺平道路。
原文摘要 · Abstract (English)
Integrating diverse visual capabilities into a unified model is a significant trend in Multimodal Large Language Models (MLLMs). Among these, the inclusion of segmentation poses a distinct set of challenges. To equip MLLMs with pixel-level segmentation abilities, prevailing methods require finetuning the model to produce specific outputs compatible with a mask decoder. This process typically alters the model's output space and compromises its intrinsic generalization, which undermines the goal of building a unified model. We introduce LENS (Leveraging kEypoiNts for MLLMs' Segmentation), a novel plug-and-play solution. LENS attaches a lightweight, trainable head to a completely frozen MLLM. By refining the spatial cues embedded in attention maps, LENS extracts keypoints and describes them into point-wise features directly compatible with the mask decoder. Extensive experiments validate our approach: LENS achieves segmentation performance competitive with or superior to that of retraining-based methods. Crucially, it does so while fully preserving the MLLM's generalization capabilities, which are significantly degraded by finetuning approaches. As such, the attachable design of LENS establishes an efficient and powerful paradigm for extending MLLMs, paving the way for truly multi-talented, unified models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。