36亿参数的轻量级视频理解模型,提升长视频处理效率。
TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler
- 采用视频级分组重采样机制,减少视觉令牌冗余。
- 在多个基准上超越70亿参数模型,训练仅需一天。
- 适合资源有限的研究者快速实验视频理解任务。
视频行为识别与场景理解是多模态智能的基础任务,广泛应用于实际场景。尽管大型多模态模型(LMMs)在视频理解方面取得显著进展,但多数开源模型依赖超过70亿参数并需大规模数据训练,资源消耗大,难以普及。此外,轻量化模型在处理长视频序列和时序理解方面仍存挑战。本文提出TinyLLaVA-Video,一个约36亿参数的轻量级视频理解模型。核心创新是视频级分组重采样器,该机制在视频层面显著减少并控制视觉令牌数量。相比传统图像级重采样,新方法有效缓解冗余问题,增强时序理解能力,从而提升视频任务性能。TinyLLaVA-Video仅需8块A100-40G GPU训练一天,即在多个基准上超越多个70亿参数模型。代码与权重已公开于https://github.com/ZhangXJ199/TinyLLaVA-Video。
原文摘要 · Abstract (English)
Video behavior recognition and scene understanding are fundamental tasks in multimodal intelligence, serving as critical building blocks for numerous real-world applications. Through large multimodal models (LMMs) have achieved remarkable progress in video understanding, most existing open-source models rely on over 7B parameters and require large-scale datasets for training, making them resource-intensive and inaccessible to many researchers. Furthermore, lightweight models face persistent challenges in effectively processing long visual sequences and temporal understanding. In this work, we introduce TinyLLaVA-Video, a lightweight yet powerful video understanding model with approximately 3.6B parameters. The cornerstone of our design is the video-level group resampler, a novel mechanism that significantly reduces and controls the number of visual tokens at the video level. Unlike traditional image-level resampler, our approach effectively mitigates redundancy while enhancing temporal comprehension, leading to improved performance on video-based tasks. In addition, TinyLLaVA-Video demonstrates exceptional efficiency, requiring only one day of training on 8 A100-40G GPUs. It surpasses several existing 7B-parameter models on multiple benchmarks. We believe this work provides a valuable foundation for future research on lightweight video understanding models. The code and weights is available at https://github.com/ZhangXJ199/TinyLLaVA-Video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。