arXiv:2512.23483cs.CV2025-12被引 12

TV-RAG通过时间衰减与语义熵加权,提升长视频理解与检索效果。

TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding

  • 引入时间衰减检索模块,根据时间偏移调整查询匹配度。
  • 用语义熵筛选关键帧,减少冗余并保持信息代表性。
  • 无需训练即可适配任意大模型,适合长视频任务研究者。

大型视频语言模型(LVLMs)在多媒体AI研究中迅速兴起,但在处理长视频时面临挑战:其时间窗口狭窄,难以捕捉长时间跨度内的细粒度语义变化。主流文本检索方法依赖表面词汇重叠,忽视视觉、音频与字幕间的丰富时间关联。为此,我们提出无需训练的TV-RAG框架,融合时间对齐与熵引导语义,提升长视频推理能力。该框架包含两个核心机制:(i) 时间衰减检索模块,在相似性计算中引入显式时间偏移,按真实多模态上下文排序查询;(ii) 语义熵加权关键帧采样器,选取均匀分布且信息密集的帧,降低冗余同时保留代表性。通过整合时间与语义信号,TV-RAG实现无需微调的双层推理流程,可无缝嵌入任意LVLM。实验表明,该系统在Video-MME、MLVU和LongVideoBench等基准上持续超越多数主流基线,验证了其有效性。代码已开源:https://github.com/AI-Researcher-Team/TV-RAG。

原文摘要 · Abstract (English)

Large Video Language Models (LVLMs) have rapidly emerged as the focus of multimedia AI research. Nonetheless, when confronted with lengthy videos, these models struggle: their temporal windows are narrow, and they fail to notice fine-grained semantic shifts that unfold over extended durations. Moreover, mainstream text-based retrieval pipelines, which rely chiefly on surface-level lexical overlap, ignore the rich temporal interdependence among visual, audio, and subtitle channels. To mitigate these limitations, we propose TV-RAG, a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning. The framework contributes two main mechanisms: \emph{(i)} a time-decay retrieval module that injects explicit temporal offsets into the similarity computation, thereby ranking text queries according to their true multimedia context; and \emph{(ii)} an entropy-weighted key-frame sampler that selects evenly spaced, information-dense frames, reducing redundancy while preserving representativeness. By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning. The resulting system offers a lightweight, budget-friendly upgrade path and consistently surpasses most leading baselines across established long-video benchmarks such as Video-MME, MLVU, and LongVideoBench, confirming the effectiveness of our model. The code can be found at https://github.com/AI-Researcher-Team/TV-RAG.

长视频理解多模态检索无训练架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。