解决视觉语言模型处理长视频时采样难的问题。
Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?
- 提出分层推理采样框架，先定位关键线索再局部密集采样。
- 在高密度采样需求任务上，用更少帧达到更好效果。
- 构建首个针对长视频采样的基准评测体系，适合视频理解研究者。
大型视觉语言模型（LVLMs）显著推进了视频理解的发展，但高效处理长视频仍面临“采样困境”：低密度采样易遗漏关键信息，高密度采样则带来冗余。为此，我们提出LSDBench，首个专门评估LVLM在长视频任务中表现的基准，通过构建高必要采样密度（NSD）问题来衡量模型所需最低采样密度。该基准聚焦短时密集动作，严格检验模型的采样策略。为应对高NSD问题，我们提出一种新的推理驱动分层采样（RHS）框架，结合全局关键线索定位与局部密集采样实现精准推理。同时，设计轻量级语义引导帧选择器，优先选取信息丰富帧，使RHS在大幅减少采样帧数的前提下保持或超越现有性能。我们的基准与评测代码已开源：https://github.com/dvlab-research/LSDBench。
原文摘要 · Abstract (English)
The rise of Large Vision-Language Models (LVLMs) has significantly advanced video understanding. However, efficiently processing long videos remains a challenge due to the ``Sampling Dilemma'': low-density sampling risks missing critical information, while high-density sampling introduces redundancy. To address this issue, we introduce LSDBench, the first benchmark designed to evaluate LVLMs on long-video tasks by constructing high Necessary Sampling Density (NSD) questions, where NSD represents the minimum sampling density required to accurately answer a given question. LSDBench focuses on dense, short-duration actions to rigorously assess the sampling strategies employed by LVLMs. To tackle the challenges posed by high-NSD questions, we propose a novel Reasoning-Driven Hierarchical Sampling (RHS) framework, which combines global localization of question-relevant cues with local dense sampling for precise inference. Additionally, we develop a lightweight Semantic-Guided Frame Selector to prioritize informative frames, enabling RHS to achieve comparable or superior performance with significantly fewer sampled frames. Together, our LSDBench and RHS framework address the unique challenges of high-NSD long-video tasks, setting a new standard for evaluating and improving LVLMs in this domain. Our benchmark and evaluation codes has been released at: https://github.com/dvlab-research/LSDBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。