arXiv:2504.10443cs.CVcs.AI2025-04被引 1

用动态时间上下文建模长视频,兼顾视觉音频信息与推理连贯性。

Multimodal Long Video Modeling Based on Temporal Dynamic Context

  • 按语义分段并用查询式变压器压缩帧间冗余信息
  • 在多个基准上超越现有方法,长视频理解准确率提升显著
  • 无需训练即可逐段推理,适合超长视频分析场景

大语言模型在视频理解方面取得突破,但受限于上下文长度和视频信息量,处理长视频仍具挑战。现有方法常在令牌压缩中丢失关键信息,且难以融合音频等多模态数据。本文提出一种基于帧间时序关系的动态长视频编码方法——时间动态上下文(TDC)。首先根据帧间相似性将视频划分为语义一致的片段,并通过视觉-音频编码器对每帧进行编码;其次设计新型时序上下文压缩器,利用查询式Transformer将视频、音频及指令文本令牌聚合为有限数量的时序上下文令牌;最后将静态帧令牌与时序上下文令牌输入大语言模型完成理解任务。针对极长视频,进一步提出无需训练的思维链策略,分段逐步提取中间答案,参与最终推理。在通用视频理解与音视频理解基准上实验验证,性能显著优于现有方法。代码与模型已开源。

原文摘要 · Abstract (English)

Recent advances in Large Language Models (LLMs) have led to significant breakthroughs in video understanding. However, existing models still struggle with long video processing due to the context length constraint of LLMs and the vast amount of information within the video. Although some recent methods are designed for long video understanding, they often lose crucial information during token compression and struggle with additional modality like audio. In this work, we propose a dynamic long video encoding method utilizing the temporal relationship between frames, named Temporal Dynamic Context (TDC). Firstly, we segment the video into semantically consistent scenes based on inter-frame similarities, then encode each frame into tokens using visual-audio encoders. Secondly, we propose a novel temporal context compressor to reduce the number of tokens within each segment. Specifically, we employ a query-based Transformer to aggregate video, audio, and instruction text tokens into a limited set of temporal context tokens. Finally, we feed the static frame tokens and the temporal context tokens into the LLM for video understanding. Furthermore, to handle extremely long videos, we propose a training-free chain-of-thought strategy that progressively extracts answers from multiple video segments. These intermediate answers serve as part of the reasoning process and contribute to the final answer. We conduct extensive experiments on general video understanding and audio-video understanding benchmarks, where our method demonstrates strong performance. The code and models are available at https://github.com/Hoar012/TDC-Video.

长视频理解多模态动态上下文推理链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。