用统一模型同时搞定图像视频的指令分割，性能超越专用模型。
InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models
- 设计端到端流水线，融合多模态大模型理解文本与视觉
- 在多个图像视频任务上超越现有分割模型和通用大模型方法
- 适合需要跨模态、跨域分割的应用场景
得益于多模态大语言模型（MLLMs）的发展，文本引导的通用图像与视频分割模型近年来取得快速进展。然而，这些方法通常针对特定领域独立开发，忽略了图像与视频任务在设定与解决方案上的共性。本文定义了图像与视频层面的指代分割与推理分割的统一问题，称为指令视觉分割（Instructed Visual Segmentation, IVS）。为此，我们提出 InstructSeg，一个基于 MLLMs 的端到端分割流水线。具体地，采用对象感知的视频感知器从参考帧中提取时空与对象信息，实现对视频的全面理解；并引入视觉引导的多粒度文本融合机制，更有效地整合全局与细节文本信息，并结合细粒度视觉引导。通过多任务与端到端训练，InstructSeg 在多种图像与视频分割任务中表现优异，单模型性能超越专用分割模型及基于 MLLM 的方法。代码已开源：https://github.com/congvvc/InstructSeg。
原文摘要 · Abstract (English)
Boosted by Multi-modal Large Language Models (MLLMs), text-guided universal segmentation models for the image and video domains have made rapid progress recently. However, these methods are often developed separately for specific domains, overlooking the similarities in task settings and solutions across these two areas. In this paper, we define the union of referring segmentation and reasoning segmentation at both the image and video levels as Instructed Visual Segmentation (IVS). Correspondingly, we propose InstructSeg, an end-to-end segmentation pipeline equipped with MLLMs for IVS. Specifically, we employ an object-aware video perceiver to extract temporal and object information from reference frames, facilitating comprehensive video understanding. Additionally, we introduce vision-guided multi-granularity text fusion to better integrate global and detailed text information with fine-grained visual guidance. By leveraging multi-task and end-to-end training, InstructSeg demonstrates superior performance across diverse image and video segmentation tasks, surpassing both segmentation specialists and MLLM-based methods with a single model. Our code is available at https://github.com/congvvc/InstructSeg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。