根据问题动态选帧，提升视频大模型理解精度。
Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs
- 基于文本图像匹配网络，用贪心采样选关键帧。
- 支持多分辨率处理，可处理更长视频而不超算力。
- 无需训练，插件式部署，适配多种视频任务。
多模态大语言模型在视觉理解任务中表现优异，但在视频理解方面仍面临数据量大、时序复杂等挑战。现有视频大模型采用均匀抽帧，难以有效捕捉与查询相关的时空关键信息。本文提出Q-Frame，一种面向视频内容和查询的自适应帧选择与多分辨率调整方法。Q-Frame采用无需训练的即插即用策略，基于CLIP等文本图像匹配网络，利用Gumbel-Max技巧实现高效帧选择，使视频大模型在不超出计算限制的前提下处理更多帧，从而保留关键时空信息。我们在MLVU、LongVideoBench和Video-MME等多个基准数据集上进行了大量实验，验证了Q-Frame在各类视频理解任务中的优越性。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated significant success in visual understanding tasks. However, challenges persist in adapting these models for video comprehension due to the large volume of data and temporal complexity. Existing Video-LLMs using uniform frame sampling often struggle to capture the query-related crucial spatiotemporal clues of videos effectively. In this paper, we introduce Q-Frame, a novel approach for adaptive frame selection and multi-resolution scaling tailored to the video's content and the specific query. Q-Frame employs a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, utilizing the Gumbel-Max trick for efficient frame selection. Q-Frame allows Video-LLMs to process more frames without exceeding computational limits, thereby preserving critical temporal and spatial information. We demonstrate Q-Frame's effectiveness through extensive experiments on benchmark datasets, including MLVU, LongVideoBench, and Video-MME, illustrating its superiority over existing methods and its applicability across various video understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。