让视频理解智能体主动寻找关键证据,省时省力还更准。
Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding
- 用迭代计划-观察-反思机制,只看关键帧,不浪费算力。
- 在五个数据集上平均准确率提升5.7%,推理时间仅需18.4%。
- 适合需要高效长视频理解的智能体系统,如AI助手、自动驾驶。
长视频理解(LVU)因关键线索稀疏分散于冗余内容中而困难重重。现有代理框架依赖与查询无关的通用描述器,导致大量计算浪费且模糊时空细节。受主动感知理论启发,我们提出主动视频感知(AVP),将视频视为可交互环境,直接从像素中获取紧凑的查询相关证据。具体而言,AVP通过多模态大模型代理执行迭代的计划-观察-反思流程:规划者提出针对性视频操作,观察者执行并提取带时间戳的证据,反思者评估证据是否足以回答问题,若不足则继续观察。在五个LVU基准测试中,AVP取得最高整体准确率,相比最优代理方法平均准确率提升5.7%,推理时间仅为18.4%,输入令牌消耗仅12.4%。
原文摘要 · Abstract (English)
Long video understanding (LVU) is challenging because answering real-world queries often depends on sparse, temporally dispersed cues buried in hours of mostly redundant and irrelevant content. While agentic pipelines improve video reasoning capabilities, prevailing frameworks rely on a query-agnostic captioner to perceive video information, which wastes computation on irrelevant content and blurs fine-grained temporal and spatial information. Motivated by active perception theory, we argue that LVU agents should actively decide what, when, and where to observe, and continuously assess whether the current observation is sufficient to answer the query. We present Active Video Perception (AVP), an evidence-seeking framework that treats the video as an interactive environment and acquires compact, queryrelevant evidence directly from pixels. Concretely, AVP runs an iterative plan-observe-reflect process with MLLM agents. In each round, a planner proposes targeted video interactions, an observer executes them to extract time-stamped evidence, and a reflector evaluates the sufficiency of the evidence for the query, either halting with an answer or triggering further observation. Across five LVU benchmarks, AVP achieves highest overall accuracy with significant improvements. Notably, AVP outperforms the best agentic method by 5.7% in average overall accuracy while only requires 18.4% inference time and 12.4% input tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。