让大模型更懂视频，通过重要性筛选压缩视觉记忆，提速34%。
REEF: Relevance-Aware and Efficient LLM Adapter for Video Understanding
- 用可微分Top-K筛选关键时空片段，保留语义相关记忆。
- 在4个数据集上达到领先或相当性能，计算开销降低34%。
- 适合需要高效视频理解的场景，如长视频分析、实时应用。
将视觉模型融入大语言模型（LLMs）已成为构建多模态基础模型的重要方向，尤其在视频理解领域备受关注。现有方法通常使用记忆库处理未剪辑视频以实现视频级理解，但普遍采用基于相似性的贪心策略压缩视觉记忆，可能忽略单个令牌的上下文重要性。为此，我们提出一种高效的LLM适配器，用于未剪辑视频的视频级理解，强调时空令牌的上下文相关性。该框架利用评分网络选择性压缩视觉记忆库，并根据相关性过滤空间令牌，采用可微分Top-K算子实现端到端训练。在三个核心视频级理解任务——未剪辑视频分类、视频问答和视频字幕生成——上，我们的方法在四个大规模数据集上取得竞争性或更优表现，同时计算开销降低最高达34%。代码即将开源。
原文摘要 · Abstract (English)
Integrating vision models into large language models (LLMs) has sparked significant interest in creating vision-language foundation models, especially for video understanding. Recent methods often utilize memory banks to handle untrimmed videos for video-level understanding. However, they typically compress visual memory using similarity-based greedy approaches, which can overlook the contextual importance of individual tokens. To address this, we introduce an efficient LLM adapter designed for video-level understanding of untrimmed videos that prioritizes the contextual relevance of spatio-temporal tokens. Our framework leverages scorer networks to selectively compress the visual memory bank and filter spatial tokens based on relevance, using a differentiable Top-K operator for end-to-end training. Across three key video-level understanding tasks$\unicode{x2013}$ untrimmed video classification, video question answering, and video captioning$\unicode{x2013}$our method achieves competitive or superior results on four large-scale datasets while reducing computational overhead by up to 34%. The code will be available soon on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。