让AI理解长达数周的视角视频,实现跨时间、多模态推理。
Agentic Very Long Video Understanding
- 基于实体场景图构建可长期追踪的视觉认知框架
- 在EgoLifeQA上达57.5%准确率,超越现有方法
- 适合需要长时序理解的智能助手与可穿戴设备应用
随全天候可穿戴设备(如智能眼镜)兴起,个人AI助手需具备超越短时事件的连续上下文理解能力,涵盖数天甚至数周的自我视角视频流。现有方法受限于有限上下文窗口,难以对超长视频流进行组合式、多跳推理。本文提出EGAgent,一种以实体场景图为核心的增强型代理框架,动态追踪人物、地点、物体及其关系。系统配备规划代理,支持结构化搜索、多模态检索(视觉+音频),实现跨时空、多模态一致的推理。在EgoLifeQA和Video-MME(Long)数据集上的实验表明,该方法在EgoLifeQA上达到57.5%的准确率,为当前最优;在Video-MME(Long)上取得74.1%的竞争力表现。代码已开源。
原文摘要 · Abstract (English)
The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video. Achieving this vision requires advances in long-horizon video understanding, where systems must interpret and recall visual and audio information spanning days or even weeks. Existing methods, including large language models and retrieval-augmented generation, are constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams. In this work, we address these challenges through EGAgent, an enhanced agentic framework centered on entity scene graphs, which represent people, places, objects, and their relationships over time. Our system equips a planning agent with tools for structured search and reasoning over these graphs, as well as hybrid visual and audio search capabilities, enabling detailed, cross-modal, and temporally coherent reasoning. Experiments on the EgoLifeQA and Video-MME (Long) datasets show that our method achieves state-of-the-art performance on EgoLifeQA (57.5%) and competitive performance on Video-MME (Long) (74.1%) for complex longitudinal video understanding tasks. Code is available at https://github.com/facebookresearch/egagent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。