arXiv:2606.17798cs.CVcs.AI2026-06

LiveStarPro让视频流能实时理解并长期记忆,主动响应不卡顿。

LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams

论文配图:LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams
图 1 · 摘自论文原文
  • 用单遍困惑度验证决定何时回复,无需静音标记。
  • 在长视频流中提升语义准确率28.9%,响应延迟降低18.2%。
  • 适合需要长时间交互的实时视频分析场景,如直播监控。

尽管视频大语言模型(Video-LLMs)取得显著进展,现有在线架构仍难以同时处理连续视频流、自主判断响应时机,并保留长期上下文记忆。这些问题削弱了实时响应能力,导致长时间交互中的严重遗忘。本文提出LiveStarPro,一种面向长时视频流的主动理解系统。其设计基于三个互补组件:第一是流式验证解码(SVeD),通过单遍困惑度验证识别合适响应时机,摆脱对显式静音标记的依赖;第二是流式因果注意力掩码(SCAM),强化可变长度流上的视频-语言增量对齐;第三是树状分层记忆(TSHM),将淘汰的历史信息组织成事件链,实现对几乎无界视频流的高效检索。为支持真实在线环境下的全面评估,我们进一步构建了OmniStarPro——一个涵盖15种真实场景、支持小时级流的大型基准。大量实验表明,LiveStarPro持续优于现有方法,在语义正确性上提升28.9%,响应时机误差减少18.2%,其流式键值缓存还带来1.58倍的推理速度提升。代码与模型已开源。

原文摘要 · Abstract (English)

Despite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory. These obstacles undermine real-time responsiveness and cause severe forgetting throughout prolonged interactions. In this work, we introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams. The design of LiveStarPro rests on three complementary components. The first component is Streaming Verification Decoding (SVeD), an inference framework that identifies the appropriate response timing through single-pass perplexity verification, thereby eliminating the dependency on explicit silence tokens. The second component is Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams. The third component is Tree-Structured Hierarchical Memory (TSHM), a recursive memory architecture that organizes evicted historical information into event chains and consequently enables efficient retrieval from effectively unbounded video streams. To facilitate a comprehensive evaluation under realistic online conditions, we further present OmniStarPro, a large-scale benchmark that spans 15 diverse real-world scenarios and that extends to hour-scale streams for the assessment of long-term recall. Extensive experiments demonstrate that LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error, while its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching. The model and the code are publicly available at https://github.com/sotayang/LiveStarPro.

视频理解长时记忆实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。