用记忆图与叙事链实现超长视频的跨模态时序理解,无需训练即可大幅提升推理能力。
Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning

- 构建多模态记忆图与叙事链,统一视觉、语义和事件信息
- 在多个超长视频数据集上超越现有最佳系统5.9~10.1分
- 适合需要长期视频推理的智能体任务,如监控、自述视频分析
理解如第一人称记录、实时流或持续数天至数周的监控视频仍具挑战。当前多模态大模型即使具备百万词上下文窗口,帧预算也仅覆盖数十分钟密集采样的视频,大部分证据在推理前已被丢弃。虽有记忆增强与智能体方法提升规模处理能力,但其检索仍分散于各模态,缺乏跨越数日甚至数周的长程叙事摘要。本文提出无需训练的MAGIC-Video框架,基于包含六种类型边的多模态记忆图与交错的叙事链:记忆图通过整合情景、语义与视觉内容实现跨模态检索,叙事链则提炼长期实体传记与重复活动事件。推理时,智能体循环结合图检索与叙事事实注入,单一流水线覆盖超长视频的模态与时间维度。在EgoLifeQA、Ego-R1与MM-Lifelong三个基准上,MAGIC-Video持续优于强基线,分别领先先前最佳智能体系统10.1、7.4与5.9分。代码已开源。
原文摘要 · Abstract (English)
Understanding ultra-long videos such as egocentric recordings, live streams, or surveillance footage spanning days to weeks, remains a challenge. For current multimodal LLMs: even with million-token context windows, frame budgets cover only tens of minutes of densely sampled video, and most evidence is discarded before inference begins. Memory-augmented and agentic approaches help with scale, but their retrieval remains fragmented across modalities and lacks long-range narrative summaries that span days or weeks. We propose \textbf{MAGIC-Video}, a training-free framework built around a multimodal memory graph with interleaved narrative chain: the graph unifies episodic, semantic, and visual content through six typed edges and supports cross-modal retrieval, while the chain distils long-horizon entity biographies and recurring activity events. At inference time, an agentic loop interleaves graph retrieval with narrative fact injection, covering both the modality and time dimensions of ultra-long video in a single retrieval pipeline. On EgoLifeQA, Ego-R1 and MM-Lifelong, MAGIC-Video consistently outperforms strong general-purpose, long-video, and agentic baselines, with gains of 10.1, 7.4, and 5.9 points over the prior best agentic system on each benchmark. Code is available at https://github.com/lijiazheng0917/MAGIC-video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。