用动态采样提升视频理解,让模型更高效地抓重点帧。
LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMs

- 基于查询感知的确定性点过程,按重要性选帧
- 在有限视觉令牌下提升2.5分,高预算下也增1.6分
- 无需训练、适配多种模型,适合资源受限场景
多模态大语言模型进行视频理解时,需在长而冗余的视频中选取关键帧,但受限于视觉令牌预算。现有方法常依赖均匀采样、逐帧评分、分块选择或代理探索,或忽略全局依赖,或引入高开销。本文提出LDDR(线性DPP-based动态分辨率采样),一种无需训练、即插即用且预算感知的视频帧采样框架。LDDR在任务相关的特征空间中执行查询感知的确定性点过程(DPP)帧选择,在标准DPP基线上实现3倍运行速度提升。进一步引入组DPP重要性度量,指导帧保留与动态分辨率分配,为信息丰富、非冗余帧分配更多令牌,对低效帧进行降采样或丢弃。在涵盖短、中、长视频的四个基准上,LDDR持续优于次优基线,在预算受限条件下提升2.5分,在高预算下提升1.6分。这些改进在多个MLLM骨干模型(含开源与闭源)上均一致有效。定性分析表明,相关帧被准确选择并分配更高预算,显著提升视频理解能力。
原文摘要 · Abstract (English)
Video understanding in multimodal large language models requires selecting informative frames from long, redundant videos under limited visual-token budgets. Existing methods often rely on uniform sampling, point-wise relevance scoring, chunk-wise selection, or agentic exploration, which either miss global dependencies or introduce substantial overhead. We propose LDDR (Linear DPP-Based Dynamic Resolution), a training-free, plug-and-play, and budget-aware video frame sampling framework. LDDR performs query-aware Determinantal Point Process (DPP) frame selection in a task-conditioned feature space, achieving a 3x runtime speedup over standard DPP baselines. It further introduces a Group DPP importance metric to guide frame retention and dynamic resolution allocation, assigning more tokens to informative, non-redundant frames while downscaling or pruning less useful ones. Across four video benchmarks spanning short-, medium-, and long-range videos, LDDR consistently outperforms the next-best baselines, achieving gains of 2.5 points under budget-constrained settings and 1.6 points in high-budget scenarios. These improvements are consistently observed across multiple MLLM backbones, including both open- and closed-source models. Qualitative analysis confirms that relevant frames are selected and allocated a higher budget, facilitating improved video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。