arXiv:2412.10471cs.CVcs.AI2024-12ICCV被引 64

用自探索智能体高效理解长视频,省去大量采帧和外部反馈。

VCA: Video Curious Agent for Long Video Understanding

  • 基于视觉语言模型构建自探索智能体,用树状搜索策略选关键片段。
  • 在多个长视频基准上表现更优,计算开销远低于传统采帧方法。
  • 适合需要低延迟、高效率长视频理解的场景,如监控分析、教育视频

长视频理解因时间复杂度高、信息密度低而面临挑战。现有方法通过大量采帧或引入大模型辅助工具,但计算成本高昂。本文提出一种基于视觉语言模型的好奇驱动视频智能体VCA,具备自主探索能力。VCA不直接采帧,而是采用树状搜索结构探索视频片段并收集关键帧;不依赖外部反馈,而是利用视觉语言模型自生成内在奖励引导探索,从而高效捕捉推理所需核心信息。在多个长视频基准上的实验表明,该方法在效果与效率上均优于现有方法。

原文摘要 · Abstract (English)

Long video understanding poses unique challenges due to their temporal complexity and low information density. Recent works address this task by sampling numerous frames or incorporating auxiliary tools using LLMs, both of which result in high computational costs. In this work, we introduce a curiosity-driven video agent with self-exploration capability, dubbed as VCA. Built upon VLMs, VCA autonomously navigates video segments and efficiently builds a comprehensive understanding of complex video sequences. Instead of directly sampling frames, VCA employs a tree-search structure to explore video segments and collect frames. Rather than relying on external feedback or reward, VCA leverages VLM's self-generated intrinsic reward to guide its exploration, enabling it to capture the most crucial information for reasoning. Experimental results on multiple long video benchmarks demonstrate our approach's superior effectiveness and efficiency.

视频理解智能体自探索高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。