轻量级视频理解框架,用单次推理实现快速响应。
Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

- 双状态设计:全局脚本+参数化隐状态,一次前向传播构建上下文
- 相比M3-Agent提升2.4%准确率,速度加快12.1倍,显存效率提升2.6倍
- 适合需要低延迟、高效率的视频代理系统,可增强现有多模态大模型
智能体视频理解通过长期记忆自主处理连续的长时程多模态流。然而,现有先进视频代理依赖‘侦探式’迭代推理进行动作控制(如搜索)与证据聚合,带来高昂开销与延迟。本文认为,这种重推理主要源于检索中缺乏全局上下文与语义错位。为此提出Light-Omni,一种具反射能力的轻量级多模态代理框架。其通过双重上下文状态,在单次前向传播中即时构建所需上下文:首先维护一个有限大小的多模态脚本,从情景记忆持续整合,作为全局上下文;其次在该全局上下文驱动下生成参数化隐状态,直接驱动自主行为并生成检索嵌入,延迟极低。得益于这一耦合设计,Light-Omni实现语义对齐的检索与反射式响应,无需迭代推理。大量实验验证其有效性,显著优于M3-Agent,平均准确率提升2.4%,速度提升12.1倍,GPU内存效率提升2.6倍。此外,其可作为记忆系统提升现有多模态大模型的性能与效率。
原文摘要 · Abstract (English)
Agentic video understanding equips models with long-term memory to autonomously process and respond to continuous, long-horizon multimodal streams. However, advanced video agents often rely on ``detective-style'' iterative reasoning for action control (e.g., $\mathtt{search}$) and evidence aggregation, incurring prohibitive costs and latency. We argue that such heavy reasoning primarily compensates for the lack of global context and semantic misalignment in retrieval. This paper introduces Light-Omni, a multimodal agent framework for reflexive and lightweight video understanding. It achieves this through dual contextual states that instantly build the required context in a single forward pass. First, we maintain a global state, a finite-sized multimodal script continuously consolidated from episodic memory, serving as the global context for Light-Omni. Through hierarchical merging, it preserves recent details while summarizing past events. Second, conditioned on this global context, we generate a parametric latent state that directly drives autonomous actions and produces retrieval embeddings, with minimal latency. Benefiting from this coupled design, Light-Omni achieves semantically aligned retrieval and reflexive responses while avoiding iterative reasoning. Extensive experiments validate the effectiveness of Light-Omni across multiple video benchmarks. Notably, it outperforms M3-Agent with an average 2.4% accuracy gain, a 12.1$\times$ speedup, and a 2.6$\times$ improvement in GPU memory efficiency. Furthermore, it serves as a memory system to enhance both the performance and efficiency of existing MLLMs. Project page: https://clare-nie.github.io/Light-Omni.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。