提出多模态记忆机制,让模型能长期理解视频内容。
WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
- 构建文本、语义、视觉三类动态记忆,支持多时序粒度检索
- 在五个长视频问答任务上平均提升8.4%性能,超越现有方法
- 适合需要长期视频推理的场景,如纪录片分析、监控视频理解
视频大语言模型在短片段理解上表现优异,但处理小时级甚至天级长视频仍面临上下文容量有限和视觉细节丢失的问题。现有基于文本摘要的记忆方法依赖文字信息,难以利用复杂场景中的视觉证据,且固定时间尺度的检索限制了对不同持续事件的捕捉能力。为此,我们提出WorldMM,一种新型多模态记忆代理,通过构建文本、语义和视觉三类互补记忆实现动态存储与检索。其中,情景记忆覆盖多时间尺度的事实事件,语义记忆持续更新高层概念知识,视觉记忆保留场景细节。推理时,自适应检索代理根据问题需求迭代选择最相关记忆源,并结合多时间粒度信息,直至确认已获取足够信息。在五个长视频问答基准测试中,WorldMM显著优于现有基线,平均性能较之前最优方法提升8.4%,验证了其在长视频推理中的有效性。
原文摘要 · Abstract (English)
Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。