arXiv:2602.12641cs.NIcs.AI2026-02

为AI视频助手优化实时通信,提升响应速度与准确率。

Artic: AI-oriented Real-time Communication for MLLM Video Assistant

  • 根据AI理解能力动态调节码率,预留带宽应对波动
  • 仅传输对回答关键的视频区域,低码率下保持高精度
  • 首个评估视频质量下降对AI理解影响的基准测试

AI视频助手作为实时通信新范式,使云端部署的多模态大语言模型(MLLM)能像真人一样交互。然而当前RTC框架与之存在根本性不匹配,导致体验质量下降和网络压力增大。实测表明现有系统会出现延迟突增和准确率下降。为此,我们提出Artic——面向MLLM视频助手的AI导向实时通信框架,推动从“人看视频”到“AI理解视频”的转变。具体包括:(1) 响应能力感知的自适应码率,利用MLLM精度饱和特性主动限制码率,预留带宽以降低延迟;(2) 零开销上下文感知流媒体,将有限码率分配给对回答最关键的视频区域,在超低码率下维持精度;(3) 退化视频理解基准(DeviBench),首个评估RTC引发的视频退化对MLLM准确性影响的基准。基于真实上行链路数据的原型实验显示,相比现有方法,Artic提升准确率15.12%,降低延迟135.31毫秒。代码与基准将开源于https://github.com/pku-netvideo/DeViBench。

原文摘要 · Abstract (English)

AI Video Assistant emerges as a new paradigm for Real-time Communication (RTC), where one peer is a Multimodal Large Language Model (MLLM) deployed in the cloud. This makes interaction between humans and AI more intuitive, akin to chatting with a real person. However, a fundamental mismatch exists between current RTC frameworks and AI Video Assistants, stemming from the drastic shift in Quality of Experience (QoE) and more challenging networks. Measurements on our production prototype also confirm that current RTC fails, causing latency spikes and accuracy drops. To address these challenges, we propose Artic, an AI-oriented RTC framework for MLLM Video Assistants, exploring the shift from "humans watching video" to "AI understanding video." Specifically, Artic proposes: (1) Response Capability-aware Adaptive Bitrate, which utilizes MLLM accuracy saturation to proactively cap bitrate, reserving bandwidth headroom to absorb future fluctuations for latency reduction; (2) Zero-overhead Context-aware Streaming, which allocates limited bitrate to regions most important for the response, maintaining accuracy even under ultra-low bitrates; and (3) Degraded Video Understanding Benchmark, the first benchmark evaluating how RTC-induced video degradation affects MLLM accuracy. Prototype experiments using real-world uplink traces show that compared with existing methods, Artic significantly improves accuracy by 15.12% and reduces latency by 135.31 ms. We will release the benchmark and codes at https://github.com/pku-netvideo/DeViBench.

实时通信AI助手视频理解码率控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。