arXiv:2505.24158cs.CV2025-05被引 10

用关键帧与叙事串联,让大模型更高效理解长视频。

Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders

  • 通过优化相关性与多样性,智能挑选关键帧
  • 插入非关键帧生成的文本叙事,减少时间断层
  • 无需训练,即插即用,适配主流多模态模型

将多模态大语言模型(MLLMs)用于长视频理解面临挑战:视频帧数多(视觉标记量大)与语言模型上下文长度有限之间的矛盾。传统均匀采样易选中无关内容,而对数千帧进行微调则计算成本高。本文提出一种即插即用的模块Nar-KFC,实现高效长视频感知。该模块包含两个协同步骤:首先将关键帧选择建模为整数二次规划问题,联合优化查询相关性与帧间多样性,并设计定制贪心搜索以降低计算复杂度;其次,为缓解稀疏关键帧采样带来的时序断裂,利用现成字幕生成器从非关键帧生成文本叙事,按真实时间顺序插入关键帧之间,形成连贯紧凑的表示。Nar-KFC是一种时空与内容感知的压缩策略,增强视觉与文本模态互补。在多个长视频基准测试中,显著提升主流MLLM性能。代码将公开。

原文摘要 · Abstract (English)

Employing Multimodal Large Language Models (MLLMs) for long video understanding remains a challenging problem due to the dilemma between the substantial number of video frames (i.e., visual tokens) versus the limited context length of language models. Traditional uniform sampling often leads to selection of irrelevant content, while post-training MLLMs on thousands of frames imposes a substantial computational burden. In this paper, we propose threading keyframes with narratives (Nar-KFC), a plug-and-play module to facilitate effective and efficient long video perception. Nar-KFC generally involves two collaborative steps. First, we formulate the keyframe selection process as an integer quadratic programming problem, jointly optimizing query-relevance and frame-diversity. To avoid its computational complexity, a customized greedy search strategy is designed as an efficient alternative. Second, to mitigate the temporal discontinuity caused by sparse keyframe sampling, we further introduce interleaved textual narratives generated from non-keyframes using off-the-shelf captioners. These narratives are inserted between keyframes based on their true temporal order, forming a coherent and compact representation. Nar-KFC thus serves as a temporal- and content-aware compression strategy that complements visual and textual modalities. Experimental results on multiple long-video benchmarks demonstrate that Nar-KFC significantly improves the performance of popular MLLMs. Code will be made publicly available.

长视频理解关键帧多模态模型文本叙事

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。