用分层图记忆与智能体检索，让视觉语言模型读懂小时级长视频。
MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism
- 将感知与推理解耦，通过分层图记忆逐步构建语义抽象结构。
- 仅用2%的上下文窗口，准确率比基线高12.5点，接近人类水平（差3.7分）。
- 适合需要长视频理解的智能体系统，尤其关注逻辑推理能力提升。
当前视觉语言模型难以处理小时级长视频，因全序列处理导致令牌爆炸和注意力稀释。为此，我们提出MemDreamer，通过解耦感知与推理，将长视频理解转化为智能体探索过程。该插件式框架增量流式输入视频，构建分层图记忆——一种自顶向下的三层语义抽象架构，核心为捕捉时空与因果关系的基础图。推理阶段，模型采用工具增强的智能体检索机制，通过观察-推理-行动循环，在层级间导航、搜索节点、遍历逻辑边。实验表明，MemDreamer在四个主流基准上达最优性能，与人类专家差距缩小至3.7分；仅需全上下文2%的推理窗口，便实现12.5分的绝对准确率提升。统计分析还揭示了视觉语言模型在逻辑推理与长视频理解间的强线性正相关，确立智能体能力扩展为多模态理解新范式。
原文摘要 · Abstract (English)
Current Vision-Language Models struggle with hours-long videos because processing full-length visual sequences induces prohibitive token explosion and attention dilution. To overcome this, we introduce MemDreamer to decouple perception and reasoning, shifting long-video understanding into an agentic exploration process. As a plug-and-play framework, it incrementally streams videos to construct a Hierarchical Graph Memory, a top-down three-tier architecture for semantic abstraction, anchored by a foundational graph capturing spatiotemporal and causal relations. During inference, the reasoning model employs agentic tool-augmented retrieval, navigating hierarchies, searching nodes, and traversing logical edges via an Observation-Reason-Action loop. Experiments show MemDreamer achieves SOTA results across four mainstream benchmarks, narrowing the gap with human experts to only 3.7 points. It constrains the reasoning context window to merely 2% of full-context ingestion while delivering a 12.5 point absolute accuracy gain. Furthermore, statistical analysis uncovers a strong positive linear correlation between an VLM's performance on logic reasoning and long-video understanding benchmarks, establishing agentic capability scaling as a new paradigm for multimodal comprehension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。