动态视频理解新框架，按需调度算力，省95%视觉令牌
An Efficient Streaming Video Understanding Framework with Agentic Control
- 分步决策：记忆压缩→响应判断→计算路由，逐步优化信息状态
- 在OVO-Bench达57.92分，StreamingBench达76.36分，视觉令牌减少95%-96%
- 适合实时视频分析场景，尤其对算力敏感的边缘设备
流式视频需在严格延迟约束下处理动态信息密度。现有方法多采用固定策略，如固定内存压缩或依赖单一模型，导致快速模型无法应对复杂查询，而持续运行的重型模型又违反实时性要求并过度处理简单问题。我们提出R3-Streaming（Remember, Respond, Reason），将流式视频理解建模为级联控制问题：针对每个查询，系统依次进行记忆压缩、响应就绪判断与计算路由，使下游决策基于逐步优化的信息状态。为优化该流程，引入年龄感知遗忘策略以实现高效记忆压缩；针对计算路由，提出TB-GRPO目标函数，通过目标平衡强化学习将复杂查询导向更强模型，防止模式崩溃。大量实验表明，R3-Streaming在流式多模态大模型中达到顶尖性能，在OVO-Bench上获57.92分，在StreamingBench上达76.36分，同时降低95%至96%的视觉标记使用量。
原文摘要 · Abstract (English)
Streaming video requires handling dynamic information density under strict latency budgets. Yet, existing methods typically employ static strategies, such as fixed memory compression or reliance on a single model, forcing a trade-off: fast models fail on complex queries, while always-on heavy models violate real-time constraints and overcomplicate simple queries. Rather than fixing these decisions upfront, we propose R3-Streaming (Remember, Respond, Reason), which formulates streaming video understanding as a cascaded control problem: for each query, the system compresses memory, judges response readiness, and routes computation sequentially, so that each downstream decision builds on progressively refined information states. To optimize this pipeline, we introduce an age-aware forgetting policy for memory compression, as aggressively compressing historical frames can yield substantial performance gains. For compute routing, we propose TB-GRPO, a target-balanced reinforcement learning objective that routes hard queries to a stronger model while preventing mode collapse. Extensive evaluations demonstrate that R3-Streaming achieves state-of-the-art results among streaming MLLMs, reaching 57.92 on OVO-Bench and 76.36 on StreamingBench, while reducing visual token usage by 95 to 96 percent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。