让视频大模型学会时间顺序，提升流式处理能力。
WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs
- 引入时间重建任务，轻量微调即让模型感知时间顺序。
- 推理时动态缓存历史，仅在不确定时扩展记忆，降低延迟。
- 无需改架构，适配现有模型，适合实时视频分析场景。
近期多模态大模型虽显著提升了视觉理解与推理能力，但其二次复杂度注意力机制和离线训练方式使其难以适应帧按序到达、未来信息不可见的流式场景。本文诊断出当前视频大模型的核心缺陷——时间无关性，即把视频当作无序证据集合而非因果序列，导致两类问题：时间顺序模糊，无法正确追踪事件先后；过去-当前关注盲区，难以区分当前观测与累积历史。为此提出WeaveTime框架，先通过轻量级时间重建目标（流式顺序感知增强）注入时间感知表征，仅需少量微调且无需专用流式数据。推理时采用过去-当前动态焦点缓存，根据不确定性触发粗粒度到细粒度检索，仅在必要时扩展历史记忆。该方法可无缝接入现有视频大模型，无需架构修改，在代表性流式基准上持续提升准确率并降低延迟，验证了其在严格在线、时间因果约束下的实用性。代码与权重将公开。项目页：https://zhangyl4.github.io/publications/weavetime/
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models have greatly improved visual understanding and reasoning, yet their quadratic attention and offline training protocols make them ill-suited for streaming settings where frames arrive sequentially and future observations are inaccessible. We diagnose a core limitation of current Video-LLMs, namely Time-Agnosticism, in which videos are treated as an unordered bag of evidence rather than a causally ordered sequence, yielding two failures in streams: temporal order ambiguity, in which the model cannot follow or reason over the correct chronological order, and past-current focus blindness where it fails to distinguish present observations from accumulated history. We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order. We introduce a lightweight Temporal Reconstruction objective-our Streaming Order Perception enhancement-that instills order aware representations with minimal finetuning and no specialized streaming data. At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed. Plugged into exsiting Video-LLM without architectural changes, WeaveTime delivers consistent gains on representative streaming benchmarks, improving accuracy while reducing latency. These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints. Code and weights will be made publicly available. Project Page: https://zhangyl4.github.io/publications/weavetime/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。