用视觉提示让视频模型看懂时间顺序,少帧也能准
ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting
- 用帧序号做视觉提示,帮模型感知时间连续性
- 仅用20%帧仍达稠密帧基线性能,提升时间推理能力
- 无需训练,适合轻量级视频理解任务
视频大语言模型(VideoLLMs)在多模态视频任务中表现优异。为降低处理密集视频帧的高计算成本,效率导向的方法如帧选择被广泛采用。然而,这些方法常导致需要时间推理的任务性能下降。与人类能从稀疏视觉线索推断事件进展不同,VideoLLMs 在省略中间帧后常误解时间关系。为此,我们探索视觉提示(VP)作为轻量但有效的增强时间理解方式。分析发现,为每帧添加显式序号信息可帮助模型感知时间连续性,并支持帧级引用,缓解稀疏采样序列中的位置歧义。基于此,我们提出 ViKey,一种无需训练的框架,结合视觉提示与轻量级关键词-帧映射(KFM)模块。KFM 使用帧索引作为字典式键,将文本线索关联至最相关帧,推理时提供明确的时间锚点。尽管结构简单,该方法显著提升时间推理能力,在部分数据集上仅用20%帧即可保持稠密帧基线性能。
原文摘要 · Abstract (English)
Recent advancements in Video Large Language Models (VideoLLMs) have enabled strong performance across diverse multimodal video tasks. To reduce the high computational cost of processing dense video frames, efficiency-oriented methods such as frame selection have been widely adopted. While effective at minimizing redundancy, these methods often cause notable performance drops on tasks requiring temporal reasoning. Unlike humans, who can infer event progression from sparse visual cues, VideoLLMs frequently misinterpret temporal relations when intermediate frames are omitted. To address this limitation, we explore visual prompting (VP) as a lightweight yet effective way to enhance temporal understanding in VideoLLMs. Our analysis reveals that simply annotating each frame with explicit ordinal information helps the model perceive temporal continuity. This visual cue also supports frame-level referencing and mitigates positional ambiguity within a sparsely sampled sequence. Building on these insights, we introduce ViKey, a training-free framework that combines VP with a lightweight Keyword-Frame Mapping (KFM) module. KFM leverages frame indices as dictionary-like keys to link textual cues to the most relevant frames, providing explicit temporal anchors during inference. Despite its simplicity, our approach substantially improves temporal reasoning and, on some datasets, preserves dense-frame baseline performance with as few as 20% of frames.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。