将视频理解帧率提升至16帧/秒,显著增强动态视觉捕捉能力。
Improving LLM Video Understanding with 16 Frames Per Second
- 采用16帧/秒高帧率采样并压缩每秒视觉令牌,高效保留动态特征。
- 在Video-MME和TemporalBench等基准上达到70亿参数模型最优表现。
- 适用于高速运动分析,优于GPT-4o等闭源视觉模型,支持低帧率推理。
人类视觉具有动态连续性,但当前多模态大语言模型(LLMs)在视频理解中主要依赖每秒≤2帧的静态图像特征提取,导致关键视觉信息丢失。本文提出F-16,首个面向高帧率视频理解的多模态LLM。通过将帧率提升至16帧/秒,并对每秒片段内的视觉令牌进行压缩,F-16高效捕获动态视觉特征,同时保留核心语义信息。实验表明,更高帧率显著提升多基准上的视频理解性能,为改进视频LLM提供超越模型规模或训练数据扩展的新路径。F-16在70亿参数视频LLM中于Video-MME与TemporalBench等通用及细粒度视频理解基准上达最优。尤其在高速运动任务(如篮球、足球、体操、跳水)中,其表现超越GPT-4o与Gemini-1.5-pro等先进闭源视觉模型。此外,我们提出一种新型解码方法,使F-16可在不重新训练的前提下实现高效低帧率推理。代码、模型检查点与数据将公开于https://github.com/bytedance/F-16。
原文摘要 · Abstract (English)
Human vision is dynamic and continuous. However, in video understanding with multimodal large language models (LLMs), existing methods primarily rely on static features extracted from images sampled at a fixed low frame rate of frame-per-second (FPS) $\leqslant$2, leading to critical visual information loss. In this paper, we introduce F-16, the first multimodal LLM designed for high-frame-rate video understanding. By increasing the frame rate to 16 FPS and compressing visual tokens within each 1-second clip, F-16 efficiently captures dynamic visual features while preserving key semantic information. Experimental results demonstrate that higher frame rates considerably enhance video understanding across multiple benchmarks, providing a new approach to improving video LLMs beyond scaling model size or training data. F-16 achieves state-of-the-art performance among 7-billion-parameter video LLMs on both general and fine-grained video understanding benchmarks, such as Video-MME and TemporalBench. Furthermore, F-16 excels in complex spatiotemporal tasks, including high-speed sports analysis (\textit{e.g.}, basketball, football, gymnastics, and diving), outperforming SOTA proprietary visual models like GPT-4o and Gemini-1.5-pro. Additionally, we introduce a novel decoding method for F-16 that enables highly efficient low-frame-rate inference without requiring model retraining. We will release the source code, model checkpoints, and data at \href{https://github.com/bytedance/F-16}{https://github.com/bytedance/F-16}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。