不训练模型，通过智能选帧提升大模型长视频理解能力
BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding
- 基于查询-帧相似度设计动态选帧策略
- 在Video-MME和MLVU上分别提升至56.1%和63.4%
- 适用于需高效处理长视频的场景
大视觉语言模型在视频理解任务中表现优异，但在长视频分析中受限于有限的上下文窗口。传统方法如均匀采样常将资源分配给无关内容，降低实际效果。本文提出BOLT，通过全面研究帧选择策略，在不增加训练的前提下提升大模型性能。首先，构建多源检索评估设置以更真实地检验模型表现；发现均匀采样在噪声环境下表现差，凸显选帧重要性。其次，探索基于查询-帧相似度的多种采样策略，结果表明逆变换采样效果最佳：在Video-MME上准确率从53.8%提升至56.1%，在MLVU上从58.9%提升至63.4%。代码已开源。
原文摘要 · Abstract (English)
Large video-language models (VLMs) have demonstrated promising progress in various video understanding tasks. However, their effectiveness in long-form video analysis is constrained by limited context windows. Traditional approaches, such as uniform frame sampling, often inevitably allocate resources to irrelevant content, diminishing their effectiveness in real-world scenarios. In this paper, we introduce BOLT, a method to BOost Large VLMs without additional Training through a comprehensive study of frame selection strategies. First, to enable a more realistic evaluation of VLMs in long-form video understanding, we propose a multi-source retrieval evaluation setting. Our findings reveal that uniform sampling performs poorly in noisy contexts, underscoring the importance of selecting the right frames. Second, we explore several frame selection strategies based on query-frame similarity and analyze their effectiveness at inference time. Our results show that inverse transform sampling yields the most significant performance improvement, increasing accuracy on the Video-MME benchmark from 53.8% to 56.1% and MLVU benchmark from 58.9% to 63.4%. Our code is available at https://github.com/sming256/BOLT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。