动态筛选视频帧，让大模型高效理解长视频。
DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding
- 根据问题动态选关键帧，减少冗余信息。
- 关键帧保留细节，其余帧轻量编码，节省内存。
- 适用于需要长视频理解的问答任务。
基于大模型的视频理解面临长视频中视觉与语义信息保持和内存成本之间的矛盾。现有方法受视频中重复及与答案无关帧的冗余影响，性能受限。通过统计分析现有数据集发现，冗余存在于重复帧与无关帧中，且对应帧随问题变化而变化，提示可采用动态编码平衡信息保留与令牌预算。为此，本文提出动态协同网络DynFocus，实现内存高效的视频编码：i) 动态事件原型估计（DPE）模块，动态选择对问答有意义的帧；ii) 紧凑协同编码（CCE）模块，对有意义帧进行细节化视觉编码，对其他帧进行简略感知编码。在五个公开基准上评估，结果一致表明该方法性能具有竞争力。
原文摘要 · Abstract (English)
The challenge in LLM-based video understanding lies in preserving visual and semantic information in long videos while maintaining a memory-affordable token count. However, redundancy and correspondence in videos have hindered the performance potential of existing methods. Through statistical learning on current datasets, we observe that redundancy occurs in both repeated and answer-irrelevant frames, and the corresponding frames vary with different questions. This suggests the possibility of adopting dynamic encoding to balance detailed video information preservation with token budget reduction. To this end, we propose a dynamic cooperative network, DynFocus, for memory-efficient video encoding in this paper. Specifically, i) a Dynamic Event Prototype Estimation (DPE) module to dynamically select meaningful frames for question answering; (ii) a Compact Cooperative Encoding (CCE) module that encodes meaningful frames with detailed visual appearance and the remaining frames with sketchy perception separately. We evaluate our method on five publicly available benchmarks, and experimental results consistently demonstrate that our method achieves competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。