arXiv:2603.04349cs.CV2026-03被引 2

通过图结构选帧提升长视频问答效率,推理速度更快、效果更好。

FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering

  • 用场景文本图表示替代原始帧序列,实现高效关键片段筛选
  • 在FindingDory和HourVideo上达到当前最优性能,推理时间大幅缩短
  • 无需训练的稀疏光流方法可直接从选中片段提取关键帧,适合部署

理解长视频对具身智能体至关重要,因其表现依赖于长期感知记忆的积累与利用。近年来,多模态大模型因具备自然语言理解与世界知识推理能力,被广泛用于长视频理解任务。然而,随着输入帧数增加,多模态大模型的响应质量下降且推理时间增长。因此,在使用多模态大模型进行长视频问答时,关键在于从中筛选出与问题相关的关键帧。本文提出FocusGraph框架,针对第一人称视角长视频问答任务设计关键帧选择方法。该框架采用轻量级可训练的场景-描述语言模型选择器,基于图结构的场景描述来挑选相关片段;随后采用无训练的逐块稀疏光流保留(PSFR)方法从这些片段中选出关键帧,并送入多模态大模型生成最终答案。相比现有方法,该选择器不依赖原始低分辨率帧顺序,而是基于紧凑的文本表示。实验表明,FocusGraph在FindingDory和HourVideo等挑战性基准上达到当前最优性能,同时显著降低推理时间。

原文摘要 · Abstract (English)

The ability to understand long videos is vital for embodied intelligent agents, because their effectiveness depends on how well they can accumulate, organize, and leverage long-horizon perceptual memories. Recently, multimodal LLMs have been gaining popularity for solving the long video understanding task due to their general ability to understand natural language and to leverage world knowledge. However, as the number of frames provided to an MLLM increases, the quality of its responses tends to degrade, and inference time grows. Therefore, when using MLLMs for long video understanding, a crucial step is selecting key frames from the video to answer user queries. In this work, we develop FocusGraph, a framework for keyframe selection for question answering over long egocentric videos. It leverages a lightweight trainable Scene-Caption LLM Selector that selects query-relevant clips based on their graph-based captions, and a training-free method for selecting keyframes from these clips. Unlike existing methods, the proposed Scene-Caption LLM Selector does not rely on the original sequence of low-resolution frames; instead, it operates on a compact textual representation of the scene. We then design a training-free Patch-wise Sparse-Flow Retention (PSFR) method to select keyframes from the resulting sequence of clips, which are fed into an MLLM to produce the final answer. Together, these components enable FocusGraph to achieve state-of-the-art results on challenging egocentric long-video question answering benchmarks, including FindingDory and HourVideo, while significantly reducing inference time relative to baseline approaches.

视频问答关键帧选择多模态模型推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。