LiveStar让直播视频理解实时响应更准更快,解决延迟与连贯性难题。
LiveStar: Live Streaming Assistant for Real-World Online Video Understanding

- 通过增量对齐训练和自适应解码,实现持续输入下的实时响应。
- 在五个任务上平均语义正确率提升19.5%,响应延迟降低18.1%。
- 适合需要高实时性与连贯性的直播分析、智能助手等场景。
尽管离线视频理解在视频大语言模型(Video-LLMs)上取得显著进展,现有在线Video-LLMs通常难以同时处理连续帧流并确定最优响应时机,常牺牲实时性与叙事连贯性。为此,我们提出LiveStar,首个实现持续主动响应的直播流助手,采用自适应流式解码。具体包括:(1) 支持变长视频流的增量视频-语言对齐训练,保持动态帧序列的时间一致性;(2) 响应静默解码框架,通过单次前向验证确定最佳主动响应时机;(3) 利用峰值-末端记忆压缩与流式键值缓存,在10分钟以上视频上实现1.53倍加速推理。我们还构建了OmniStar数据集,涵盖15种真实场景和5项评估任务,用于训练与评测。在三个基准上的实验表明,LiveStar在语义正确性上平均提升19.5%,响应时间差减少18.1%,所有五项任务中帧率提升12.0%。模型与数据集可访问:https://github.com/yzy-bupt/LiveStar。
原文摘要 · Abstract (English)
Despite significant progress in Video Large Language Models (Video-LLMs) for offline video understanding, existing online Video-LLMs typically struggle to simultaneously process continuous frame-by-frame inputs and determine optimal response timing, often compromising real-time responsiveness and narrative coherence. To address these limitations, we introduce LiveStar, a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding. Specifically, LiveStar incorporates: (1) a training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) a response-silence decoding framework that determines optimal proactive response timing via a single forward pass verification; (3) memory-aware acceleration via peak-end memory compression for online inference on 10+ minute videos, combined with streaming key-value cache to achieve 1.53x faster inference. We also construct an OmniStar dataset, a comprehensive dataset for training and benchmarking that encompasses 15 diverse real-world scenarios and 5 evaluation tasks for online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the-art performance, achieving an average 19.5% improvement in semantic correctness with 18.1% reduced timing difference compared to existing online Video-LLMs, while improving FPS by 12.0% across all five OmniStar tasks. Our model and dataset can be accessed at https://github.com/yzy-bupt/LiveStar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。