arXiv:2606.29445cs.CVcs.AI2026-06中稿 · ECCV

提出通用关键帧提取方法,提升模型看视频做任务的能力。

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

论文配图:Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction
图 1 · 摘自论文原文
  • 根据任务相关性和场景动态,智能筛选关键帧。
  • 在两个基准上分别提升2.0%和1.8%性能。
  • 适合研究视频理解与智能体交互的学者使用。

视频理解是多模态智能的基础能力,当前多模态大模型在视频问答(VideoQA)任务上表现优异。但现有评测主要考察模型对浅层视觉线索的感知,较少评估其能否从视频教程中学习深层知识或操作技能,并迁移到长时序智能体任务。为此,我们提出VG-GUIBench(视频引导图形界面基准),用于评估基于多模态大模型的GUI智能体是否能通过视频教程完成交互任务。进一步观察发现,模型在VideoQA与视频引导智能体任务上的表现均高度依赖有效的关键帧提取。据此,我们提出TASKER(任务驱动且场景感知的关键帧搜索器),联合考虑任务相关性与场景动态,识别高信息量帧。实验表明,TASKER在EgoSchema全集和NExT-QA数据集上分别优于最佳基线2.0%和1.8%,验证了通用关键帧提取方法在视频理解中的潜力。代码与数据已开源。

原文摘要 · Abstract (English)

Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.

视频理解关键帧提取智能体任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。