发现大模型知道事件时间却说不准,通过注意力信号修复预测。
MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues

- 利用注意力头识别模型在预填充阶段已感知到的时间区间。
- 修复生成阶段注意力偏移,使时间定位准确率提升最高3.5个mIoU。
- 无需训练或修改模型,适用于多个主流多模态大模型。
视频时间定位(VTG)旨在从无剪辑视频中精确定位查询事件的起止时间,是检验多模态大模型是否真正理解‘何时发生’的关键任务。尽管现代多模态大模型能流畅描述视频内容,其时间戳预测仍不可靠。现有方法要么需昂贵的时间标注微调,要么依赖粗略的无训练启发式策略。本文探究多模态大模型的跨模态注意力,揭示了感知与生成之间的差距:模型在预填充阶段通常已知目标时间段,但在自回归生成时注意力会偏离至视觉显著但与查询无关的片段。我们发现一组稀疏的注意力头(称作‘时间定位头’,TG-Heads)在预填充阶段集中于真实时间区间。基于此,提出一种推理时‘读取再重生成’框架:将TG-Head的注意力转换为去偏的帧级相关性信号,提取高注意力区间,并通过裁剪或注意力掩码限制视觉上下文重调模型。该方法无需参数更新或结构修改,在MiMo-VL-7B、Qwen3-VL-8B和TimeLens-8B上均在三个VTG基准上取得提升,最高达+3.5 mIoU。
原文摘要 · Abstract (English)
Video temporal grounding (VTG), which localizes the start and end times of a queried event in an untrimmed video, is a key test of whether multimodal large language models (MLLMs) understand not only what happens but also when it happens. Although modern MLLMs describe video content fluently, their timestamp predictions remain unreliable, while existing remedies either require costly post-training on temporal annotations or rely on coarse training-free heuristics. In this work, we probe the cross-modal attention of MLLMs and uncover a perception-generation gap. Our key finding is that MLLMs often know the target interval during prefill, but lose this signal when generating the final answer. In the prefill stage, a sparse set of attention heads, which we call \emph{Temporal Grounding Heads} (TG-Heads), concentrates query-to-video attention on the ground-truth interval. During autoregressive decoding, however, the answer tokens shift attention away from this interval toward visually salient but query-irrelevant segments. This observation motivates an inference-time read-then-regenerate framework. We first convert TG-Head prefill attention into a debiased frame-level relevance signal and extract the high-attention interval it highlights. We then re-invoke the MLLM with visual context restricted to this interval, using video cropping or attention masking to suppress distractors. Without parameter updates and architectural changes, our framework consistently improves MiMo-VL-7B, Qwen3-VL-8B, and TimeLens-8B on three VTG benchmarks, with gains of up to +3.5 mIoU. The project website can be found at https://ddz16.github.io/mllmsknowwhen.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。