arXiv:2605.10762cs.CVcs.AI2026-05被引 1

用推理结果动态选帧,让长视频理解更快更准。

GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs

论文配图:GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs
图 1 · 摘自论文原文
  • 通过冻结模型的输出概率构建轻量探针,动态评估每帧相关性。
  • 在Video-MME-v2上减少3.36倍计算量,准确率仅降1.6个百分点。
  • 可与不同规模问答模型搭配,无需重训即可提升性能与效率。

长视频理解在视觉语言模型中受限于数千帧的单次前向传播及二次复杂度注意力计算。常见缓解方法是在前向传播前选取少量信息帧,多采用训练无关的选择器,基于辅助编码空间相似性。但此类信号受对比预训练限制,在需要推理的任务(如否定、跨帧计数、整体摘要)上表现不佳。本文提出GridProbe,一种高效的训练无关后验探查推理范式,利用冻结的VLM自身推理结果,在答案空间评分证据,并自适应选择与问题相关的帧,实现亚二次注意力开销且几乎无精度损失。将帧排列成K×K网格,运行轻量行R与列C探针,每个探针以峰值后验概率作为条件置信度。R与C的外积生成可解释的重要性图,其偏度与峰度驱动形状自适应选择,以闭式规则替代固定帧预算M,转为按问题动态调整的有效帧数M_eff。实证显示,M_eff在未见答案情况下能追踪问题内在难度,体现测试时自适应计算。在Video-MME-v2上,GridProbe以3.36倍的计算缩减,保持平均准确率仅下降1.6个百分点;在LongVideoBench上,实现对基线的帕累托优势(+0.9个百分点,计算降至0.35倍)。由于选择器与问答模型可解耦,搭配小规模2B选择器与更强的4B或8B问答模型,相比2B全模型严格帕累托占优(平均提升+4.0个百分点,计算降至0.52倍),无需重训练。重要性图的可解释性也为行为诊断、定位与帧选择蒸馏开辟新路径。

原文摘要 · Abstract (English)

Long-video understanding in VLMs is bottlenecked by a single monolithic forward pass over thousands of frames at quadratic attention cost. A common mitigation is to first select a small subset of informative frames before the forward pass; common for training-free selectors via auxiliary encoder-space similarities. Such signals are capped by contrastive pretraining, which usually fails on reasoning-heavy queries (negation, cross-frame counting, holistic summarization). We propose GridProbe, an efficient training-free posterior-probing inference paradigm that scores evidence in answer space using a frozen VLM's own reasoning and then selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss. We arrange frames on a $K{\times}K$ grid and run lightweight row R and column C probes, where each probe reads its peak posterior as a query-conditioned confidence. The outer product of R and C yields an interpretable importance map whose skewness and kurtosis drive Shape-Adaptive Selection, a closed-form rule that reliably replaces the fixed frame budget $M$ with a per-question $M_{\mathrm{eff}}$. We show empirically that $M_{\mathrm{eff}}$ tracks intrinsic question difficulty without ever seeing the answer, a sign of test-time adaptive compute. On Video-MME-v2, GridProbe matches the monolithic baseline within $1.6$ pp Avg Acc at $3.36\times$ TFLOPs reduction, while on LongVideoBench it Pareto-dominates the baseline ($+0.9$ pp at $0.35\times$ compute). Because the selector and QA models can be decoupled, pairing a small 2B selector with a stronger 4B or 8B QA is strictly Pareto-dominant over the 2B monolithic baseline (up to $+4.0$ pp at $0.52\times$ compute, on average), with no retraining. Finally, the interpretability of the importance maps opens future avenues for behavioral diagnostics, grounding, and frame-selection distillation.

长视频理解自适应计算后验探查可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。