让视频与文字精准对齐，实现像素级视觉定位。
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
- 三组件协同：语言模型+双视觉编码器+时空解码器，通过可调适配器强化图文对齐。
- 在38000个视频问答对上训练，生成67万像素掩码，实现高精度定位。
- 适合需要精确视频理解的场景，如智能剪辑、自动驾驶视觉分析。
视频与文本之间的细粒度对齐因复杂的时空动态而极具挑战。现有视频大模型虽能处理基础对话，但在像素级定位任务上表现不佳。为此，我们提出VideoGLaMM，一种专为基于文本输入的视频像素级定位设计的大规模多模态模型。该模型通过可调节的视觉-语言（V-L）和语言-视觉（L-V）适配器，将大语言模型、强调空间与时间细节的双视觉编码器以及用于精确掩码生成的时空解码器无缝连接。模型在训练中同步视频内容的空间与时间元素与文本指令。为支持细粒度定位，我们采用半自动标注流程构建了一个多模态数据集，包含38,000个视频-问答三元组，涵盖83,000个物体和671,000个掩码。我们在三项挑战性任务上评估该模型：具身对话生成、视觉定位和指代视频分割。实验结果表明，我们的模型在所有任务中均显著优于现有方法。
原文摘要 · Abstract (English)
Fine-grained alignment between videos and text is challenging due to complex spatial and temporal dynamics in videos. Existing video-based Large Multimodal Models (LMMs) handle basic conversations but struggle with precise pixel-level grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed for fine-grained pixel-level grounding in videos based on user-provided textual inputs. Our design seamlessly connects three key components: a Large Language Model, a dual vision encoder that emphasizes both spatial and temporal details, and a spatio-temporal decoder for accurate mask generation. This connection is facilitated via tunable V-L and L-V adapters that enable close Vision-Language (VL) alignment. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. To enable fine-grained grounding, we curate a multimodal dataset featuring detailed visually-grounded conversations using a semiautomatic annotation pipeline, resulting in a diverse set of 38k video-QA triplets along with 83k objects and 671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation. Experimental results show that our model consistently outperforms existing approaches across all three tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。