提出DisCo方法，让视频视觉编码更清晰连贯
DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs
- 用概念判别器为视觉标记分配独特语义
- 通过时间聚焦校准确保帧间语义一致
- 在多个视频理解任务中表现领先，效率更高
在视频多模态大语言模型中，视觉封装将视频内容转化为适合大语言模型输入的表征标记至关重要。尽管线性投影器广泛应用，但会导致语义模糊和时间不连贯。而重采样结构虽有潜力，有效解决方案仍待探索。受重采样结构启发，我们提出DisCo，一种新型视觉封装方法，旨在生成语义分明且时间连贯的视觉标记。DisCo包含两个关键组件：(1) 视觉概念判别器（VCD）模块，通过将视觉标记与视频中的判别性概念配对，赋予其唯一语义；(2) 时间焦点校准器（TFC）模块，确保视觉标记在每一帧中对视频元素保持一致的时间关注。在多个视频多模态大语言模型框架上的大量实验表明，DisCo显著优于此前最先进的方法，在多种视频理解基准上表现优异，同时因减少语义模糊而实现更高的标记效率。
原文摘要 · Abstract (English)
In video Multimodal Large Language Models (video MLLMs), the visual encapsulation process plays a pivotal role in converting video contents into representative tokens for LLM input. While linear projectors are widely employed for encapsulation, they introduce semantic indistinctness and temporal incoherence when applied to videos. Conversely, the structure of resamplers shows promise in tackling these challenges, but an effective solution remains unexplored. Drawing inspiration from resampler structures, we introduce DisCo, a novel visual encapsulation method designed to yield semantically distinct and temporally coherent visual tokens for video MLLMs. DisCo integrates two key components: (1) A Visual Concept Discriminator (VCD) module, assigning unique semantics for visual tokens by associating them in pair with discriminative concepts in the video. (2) A Temporal Focus Calibrator (TFC) module, ensuring consistent temporal focus of visual tokens to video elements across every video frame. Through extensive experiments on multiple video MLLM frameworks, we demonstrate that DisCo remarkably outperforms previous state-of-the-art methods across a variety of video understanding benchmarks, while also achieving higher token efficiency thanks to the reduction of semantic indistinctness. The code: https://github.com/ZJHTerry18/DisCo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。