提出可自适应任务的视频帧选择方法,提升视频理解效率
HFS: Holistic Query-Aware Frame Selection for Efficient Video Understanding
- 用小语言模型生成任务相关查询向量,动态调整帧评分
- 设计可微分的集合级优化目标,同时兼顾相关性、覆盖性和冗余度
- 学生-教师协同训练,实现端到端优化,无需预设伪标签
关键帧选择本质上是一个集合级优化问题:所选子集的质量取决于帧之间的相互作用,而非单个帧的得分。现有方法存在三大局限:点级方法独立评分每帧,忽略帧间依赖;无训练集方法虽显式建模帧间关系,但选择标准固定,无法随下游任务反馈调整;可学习方法虽能数据驱动训练,却缺乏显式的可微分集合质量目标,且依赖离线生成的监督信号。为此,我们提出一种端到端可训练、任务自适应的帧选择框架。通过思维链提示引导小语言模型(SLM)提取任务相关的隐式查询向量,与多模态特征结合,实现动态、查询感知的帧评分。进一步构建连续的集合级目标函数,联合考虑相关性、覆盖性和冗余性,利用Gumbel-TopK实现可微分的集合级优化,以选出最优帧组合。最后采用学生-教师互学习策略,使学生选择器(SLM)与教师推理器(MLLM)通过KL散度对齐帧重要性分布,结合交叉熵损失,实现完全端到端优化,摆脱对静态伪标签的依赖。在Video-MME、LongVideoBench、MLVU和NExT-QA等多个基准上的实验表明,该方法显著优于现有帧选择方法。
原文摘要 · Abstract (English)
Key frame selection is essentially a set-level optimization problem: the quality of the selected subset depends on the interactions among frames, rather than the score of any single frame. Existing methods generally exhibit three major limitations. Point-wise methods score each frame independently and ignore inter-frame dependencies. Although the training-free set-level methods explicitly model the inter-frame relationships, their selection criteria are fixed and cannot be adapted through downstream task feedback. Learnable methods can leverage data-driven training; however, they lack an explicit, differentiable set-quality objective and rely on offline-generated supervision signals. To address these limitations, we propose an end-to-end trainable and task-adaptive framework for frame selection. A Chain-of-Thought prompt conditions a Small Language Model (SLM) to extract task-specific latent query vectors, which are combined with multimodal features to enable dynamic, query-aware frame scoring. We further formulate a continuous set-level objective function that jointly accounts for relevance, coverage, and redundancy, enabling differentiable set-level optimization via Gumbel-TopK for selecting optimal frame combinations. Finally, we employ a student-teacher mutual learning strategy, in which the student selector (SLM) and teacher reasoner (MLLM) are trained to align their frame-importance distributions via KL divergence. Combined with cross-entropy loss, this design enables fully end-to-end optimization, eliminating reliance on static pseudo-labels. Experiments across multiple benchmarks, including Video-MME, LongVideoBench, MLVU, and NExT-QA, demonstrate that our method significantly outperforms existing frame-selection approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。