不训练即可提升长视频理解,让选帧更精准
FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

- 通过查询相关性几何空间,统一重要性和多样性目标
- 在16~64帧预算下,关键帧选择得分提升11.0-15.3点
- 适用于各类大模型,显著提升问答准确率
多模态大语言模型(MLLM)实现了前所未有的长视频理解能力。然而,随着视频序列变长,相关内容密度急剧下降,引入过多无关帧会显著降低模型精度。本文提出FORGE(Frame Orthogonality in Relevance Geometry),一种无需训练的模型无关方法,在预训练多模态嵌入空间中构建查询相关的几何结构,将相关性与多样性统一为单一目标。在此空间中,覆盖独立相关方向的帧彼此远离,通过选取最大信息量的帧子集,可在预算内捕获多样化的查询相关内容。在Video-MME和LongVideoBench上,于16、32、64帧预算下,FORGE相比最强无训练基线,关键帧选择得分提升11.0-15.3点,关键帧召回率最高达0.415(原为0.204,提升近一倍)。该方法在问答任务中同样有效,8个开源MLLM(4B至32B参数)在所有设置下均实现精度提升,相较均匀采样最高提升8.7点,较最强基线提升5.2点。结果表明,对齐嵌入空间与查询的高维结构是推理时视频理解的可行方向。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible. However, the density of relevant content decreases sharply as video sequence length increases, and exposing the model to more irrelevant content measurably reduces its accuracy. In this paper, we address the problem of maximizing query-relevant information in a frame subset selected at inference time, without training. FORGE (Frame Orthogonality in Relevance Geometry) is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space, unifying relevance and diversity into a single objective. In this space, frames that cover independent query-relevant directions are far apart, and selecting the subset of maximum information captures diverse query-relevant content within the budget. Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified keyframe selection score by 11.0-15.3 points over the strongest training-free baseline and up to doubles keyframe recall (0.415 vs. 0.204 at K=64 on Video-MME). The gains extend to question answering, where accuracy improves in every evaluated setting across eight open-source MLLMs spanning 4B to 32B parameters, by up to 8.7 points over uniform sampling and 5.2 points over the strongest baseline. Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。