轻量级视频理解模型，实时推理速度超46词/秒
Mobile-VideoGPT: Fast and Accurate Model for Mobile Video Understanding
- 双轻量视觉编码器+小语言模型，参数少于10亿
- 关键帧筛选与视觉令牌压缩，实现46词/秒吞吐
- 适合移动端部署，比同类模型快2倍以上
视频理解模型常因计算开销大、参数多、推理慢而难以实用。为此，我们提出Mobile-VideoGPT，一个参数少于十亿的高效多模态框架。不同于传统视频大模型（LMM），该模型采用轻量双视觉编码器、高效投影器和小型语言模型（SLM），支持实时推理。为提升效率，引入基于注意力的关键帧评分机制，并设计高效令牌投影器，剔除冗余视觉信息，保留关键上下文。在六个主流视频理解基准（如MVBench、EgoSchema、NextQA、PercepTest）上评估显示，Mobile-VideoGPT-0.5B可实现每秒46个词的生成速度，在平均性能上超越现有0.5B参数模型6个百分点，参数减少40%，吞吐提升2倍以上。代码与模型已开源。
原文摘要 · Abstract (English)
Video understanding models often struggle with high computational requirements, extensive parameter counts, and slow inference speed, making them inefficient for practical use. To tackle these challenges, we propose Mobile-VideoGPT, an efficient multimodal framework designed to operate with fewer than a billion parameters. Unlike traditional video large multimodal models (LMMs), Mobile-VideoGPT consists of lightweight dual visual encoders, efficient projectors, and a small language model (SLM), enabling real-time throughput. To further improve efficiency, we present an Attention-Based Frame Scoring mechanism to select the key-frames, along with an efficient token projector that prunes redundant visual tokens and preserves essential contextual cues. We evaluate our model across well-established six video understanding benchmarks (e.g., MVBench, EgoSchema, NextQA, and PercepTest). Our results show that Mobile-VideoGPT-0.5B can generate up to 46 tokens per second while outperforming existing state-of-the-art 0.5B-parameter models by 6 points on average with 40% fewer parameters and more than 2x higher throughput. Our code and models are publicly available at: https://github.com/Amshaker/Mobile-VideoGPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。