arXiv:2504.10068cs.CVcs.AI2025-04被引 27

Mavors通过多粒度视频表示,实现长视频高效精准理解。

Mavors: Multi-granularity Video Representation for Multimodal Large Language Model

  • 将视频分块处理,用3D卷积与Transformer保持空间细节。
  • 跨块特征聚合器利用旋转位置编码建模时间依赖,提升时序连贯性。
  • 统一图像与视频理解,适合复杂运动或高分辨率视频任务。

多模态大模型在长视频理解中面临计算效率与细粒度时空模式保留之间的矛盾。现有方法(如稀疏采样、低分辨率密集采样、令牌压缩)在复杂运动或变分辨率视频中易丢失时序动态、空间细节或细微交互。为此,我们提出Mavors框架,引入多粒度视频表示,直接将原始视频内容编码为潜在表示。核心包括:1)块内视觉编码器(IVE),通过3D卷积与视觉Transformer保留高分辨率空间特征;2)块间特征聚合器(IFA),基于Transformer依赖建模与块级旋转位置编码,建立跨块时间连贯性。此外,该框架通过子图像分解将图像视为单帧视频,统一图像与视频理解。跨多样基准测试表明,Mavors在保持空间保真度与时序连续性方面显著优于现有方法,尤其在需细粒度时空推理的任务中表现突出。

原文摘要 · Abstract (English)

Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity $\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning.

视频理解多模态长视频时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。