提出Clapper方法，实现视频模型13倍压缩不丢精度
Clapper: Compact Learning and Video Representation in VLMs
- 用快慢双通道策略处理视频时序与空间信息
- 每帧仅保留61个视觉标记，压缩13倍仍保持准确率
- 适合需要高效长视频理解的场景，如智能监控
当前视觉语言模型在视频理解任务中表现卓越，但如何有效建模时间维度并平衡长短视频处理成为挑战。短视频需保留细节，长视频则需压缩信息以应对长时间上下文。然而实证发现，现有模型在视觉标记压缩至原始四分之一以下时性能急剧下降。为此，我们提出Clapper方法，采用快慢双通道策略，并引入TimePerceiver模块，在不改变原有模型结构的前提下实现高效的时空编码。该方法可将每帧视觉标记压缩至平均61个（即13倍压缩），在视频理解任务中保持高精度：VideoMME达62.0%，MLVU达69.8%，TempCompass达67.4%，且全视频视觉标记数少于6,000。代码将在主页公开。
原文摘要 · Abstract (English)
Current vision-language models (VLMs) have demonstrated remarkable capabilities across diverse video understanding applications. Designing VLMs for video inputs requires effectively modeling the temporal dimension (i.e. capturing dependencies across frames) and balancing the processing of short and long videos. Specifically, short videos demand preservation of fine-grained details, whereas long videos require strategic compression of visual information to handle extensive temporal contexts efficiently. However, our empirical analysis reveals a critical limitation: most existing VLMs suffer severe performance degradation in long video understanding tasks when compressing visual tokens below a quarter of their original visual tokens. To enable more effective modeling of both short and long video inputs, we propose Clapper, a method that utilizes a slow-fast strategy for video representation and introduces a novel module named TimePerceiver for efficient temporal-spatial encoding within existing VLM backbones. By using our method, we achieves 13x compression of visual tokens per frame (averaging 61 tokens/frame) without compromising QA accuracy. In our experiments, Clapper achieves 62.0% on VideoMME, 69.8% on MLVU, and 67.4% on TempCompass, all with fewer than 6,000 visual tokens per video. The code will be publicly available on the homepage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。