提出GRT框架,让视频模型高效理解高帧率细节。
Dense Video Understanding with Gated Residual Tokenization
- 用运动补偿跳过静态区域,实现非线性增的令牌数
- 在高帧率下准确率超大模型,且随帧率提升表现更好
- 适合需要精准时间对齐的任务,如讲课视频分析
高时间分辨率对捕捉视频中的细粒度信息至关重要。然而,当前视频大语言模型(VLLM)和基准测试多依赖低帧率采样,如均匀采样或关键帧选择,丢失了密集的时间信息。这种权衡虽避免了每帧令牌化的高开销,但导致令牌数量与视频长度呈线性增长,造成冗余计算。尽管这对缓慢变化内容有效,却难以应对讲座理解等每帧都含信息的任务。为此,我们提出稠密视频理解(DVU),通过减少令牌化时间和令牌开销,实现高帧率视频理解。同时,我们构建首个面向稠密时间推理的基准DIVE(Dense Information Video Evaluation)。为使DVU实用,提出门控残差令牌化(GRT):(1)运动补偿跨帧门控令牌化利用像素级运动估计跳过静态区域,实现令牌数与计算量的次线性增长;(2)语义场景内令牌融合将同一场景内静态区域的令牌合并,进一步减少冗余并保留动态语义。在DIVE上的实验表明,GRT优于更大规模的VLLM基线,且随帧率提升表现持续改善。结果凸显了稠密时间信息的重要性,并证明GRT可实现高效、可扩展的高帧率视频理解。
原文摘要 · Abstract (English)
High temporal resolution is essential for capturing fine-grained details in video understanding. However, current video large language models (VLLMs) and benchmarks mostly rely on low-frame-rate sampling, such as uniform sampling or keyframe selection, discarding dense temporal information. This compromise avoids the high cost of tokenizing every frame, which otherwise leads to redundant computation and linear token growth as video length increases. While this trade-off works for slowly changing content, it fails for tasks like lecture comprehension, where information appears in nearly every frame and requires precise temporal alignment. To address this gap, we introduce Dense Video Understanding (DVU), which enables high-FPS video comprehension by reducing both tokenization time and token overhead. Existing benchmarks are also limited, as their QA pairs focus on coarse content changes. We therefore propose DIVE (Dense Information Video Evaluation), the first benchmark designed for dense temporal reasoning. To make DVU practical, we present Gated Residual Tokenization (GRT), a two-stage framework: (1) Motion-Compensated Inter-Gated Tokenization uses pixel-level motion estimation to skip static regions during tokenization, achieving sub-linear growth in token count and compute. (2) Semantic-Scene Intra-Tokenization Merging fuses tokens across static regions within a scene, further reducing redundancy while preserving dynamic semantics. Experiments on DIVE show that GRT outperforms larger VLLM baselines and scales positively with FPS. These results highlight the importance of dense temporal information and demonstrate that GRT enables efficient, scalable high-FPS video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。