arXiv:2510.04428cs.CV2025-10

用智能迭代选帧让大模型高效答视频题,省时又准。

A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering

  • 用大模型深度分析问题,再逐轮筛选最有潜力的帧
  • 在多个数据集上显著提升答题准确率,计算量仅为传统方法1/10
  • 无需训练,适合想用大模型做视频问答的研究者和开发者

将视觉语言模型(VLM)有效应用于视频问答(VideoQA)的关键在于选取精炼而全面的帧集,因处理整段视频计算成本过高。现有选帧方法面临根本性权衡:依赖轻量级相似度模型(如CLIP)的方法难以捕捉复杂查询的细微差别,导致相似度评分失真,影响选帧效果;而采用VLM进行深度分析的方法虽准确,但计算开销巨大。为此,我们提出A.I.R.——一种无需训练的自适应、迭代式、基于推理的帧选择方法。利用强大VLM对复杂问题进行深层语义分析,并将其嵌入低成本的迭代循环中,每次仅处理一小批最具潜力的帧。在多个VideoQA基准上的实验表明,该方法优于现有选帧技术,显著提升基础VLM性能,且相比其他VLM方法实现显著的计算效率提升。

原文摘要 · Abstract (English)

Effectively applying Vision-Language Models (VLMs) to Video Question Answering (VideoQA) hinges on selecting a concise yet comprehensive set of frames, as processing entire videos is computationally infeasible. However, current frame selection methods face a critical trade-off: approaches relying on lightweight similarity models, such as CLIP, often fail to capture the nuances of complex queries, resulting in inaccurate similarity scores that cannot reflect the authentic query-frame relevance, which further undermines frame selection. Meanwhile, methods that leverage a VLM for deeper analysis achieve higher accuracy but incur prohibitive computational costs. To address these limitations, we propose A.I.R., a training-free approach for Adaptive, Iterative, and Reasoning-based frame selection. We leverage a powerful VLM to perform deep, semantic analysis on complex queries, and this analysis is deployed within a cost-effective iterative loop that processes only a small batch of the most high-potential frames at a time. Extensive experiments on various VideoQA benchmarks demonstrate that our approach outperforms existing frame selection methods, significantly boosts the performance of the foundation VLM, and achieves substantial gains in computational efficiency over other VLM-based techniques.

视频问答帧选择VLM推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。