TimeViper用混合架构实现超长视频理解,处理时长超1小时视频。
TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
- 融合Mamba与Transformer的混合架构,兼顾效率与表达力。
- 可处理超过10,000帧的长视频,性能媲美顶尖模型。
- 提出信息迁移模块TransV,缓解视觉令牌冗余问题。
我们提出TimeViper,一种用于解决长视频理解挑战的混合视觉语言模型。处理长视频需要高效的模型架构和对长时序上下文的有效建模能力。为此,TimeViper采用混合的Mamba-Transformer主干网络,结合状态空间模型的高效性与注意力机制的表达力。通过该设计,我们发现视觉到文本的信息聚合现象:随着大语言模型深度增加,信息逐步从视觉标记流向文本标记,导致视觉标记严重冗余。基于此观察,我们提出TransV,一个将视觉标记转移并压缩至指令标记的模块,同时保持多模态理解能力。该设计使TimeViper能够处理超过10,000帧的小时级视频。在多个基准上的大量实验表明,TimeViper在扩展帧数的同时仍具备与前沿模型竞争的性能。我们进一步分析了Mamba与Transformer层的注意力行为,为混合模型的可解释性提供了新见解。本工作是迈向开发、解析与压缩混合Mamba-Transformer架构的初步探索。
原文摘要 · Abstract (English)
We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。