Homer让大模型像人一样理解长视频,通过分层记忆和推理机制提升问答准确率。
Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning

- 构建分层级记忆结构,涵盖感知、实体到事件因果关系
- 在三个长视频基准上分别提升5.5、10.8、4.4分,超越现有方法
- 适用于多种大模型,可通用增强长视频理解能力
多模态大模型在短视频上表现优异,但在在线处理小时级长视频时受限于有限内存。现有方法要么保留缺乏语义结构的紧凑视觉表示,要么基于时间邻近性构建高层记忆,无法显式建模因果关系,导致多跳叙事推理需由LLM反复重建。本文提出Homer——一种分层在线记忆探索与推理框架。其记忆结构模拟长视频的多尺度特征,从原始感知到重复实体,再到由显式时序与因果关系连接的事件。配套的代理推理器模仿人类认知过程:定位场景、查询细节、多轮检索并生成答案,每一步均有验证与纠错机制。Homer在M3-Bench-robot、M3-Bench-web、Video-MME-Long上分别领先前最优方法5.5、10.8、4.4分,且对三种不同LLM骨干网络均持续提升,表明其具备模型无关的长视频接地检索能力。
原文摘要 · Abstract (English)
Multimodal large language models excel on short clips but struggle on hour-long videos in an online setting, where frames are processed incrementally under limited memory. Existing online methods either retain compact visual representations that lack semantic structure, or build higher-level memory stores organized around temporal proximity rather than explicit causal links, leaving multi-hop narrative reasoning to be reconstructed by the LLM at every query. We bridge this gap with \textsc{Homer}, a Hierarchical Online Memory Exploration and Reasoning framework. \textsc{Homer}'s memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations. Its agentic reasoner then explores this memory the way humans do, locating the relevant scene, looking up details, and composing the answer through multi-round memory retrieval, with a harness that verifies and corrects each step. \textsc{Homer} outperforms the previous best agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long, and consistently lifts three various LLM backbones, indicating a model-agnostic structural capability for grounded retrieval over long videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。