让AI实时视频对话更流畅，关键在智能压缩视频关键区域。
Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AI
- 根据对话重要性分配视频码率，只传关键画面
- 实测证明极低码率仍可保持AI理解准确率
- 适合研究AI通信、低延迟视频系统的开发者
AI视频聊天作为实时通信新范式，使人类与多模态大语言模型（MLLM）进行面对面交流。但MLLM推理耗时长，网络延迟成主要瓶颈。本文提出面向AI的实时通信研究方向，强调从“人类看视频”转向“AI理解视频”的需求转变。通过原型测试发现，超低码率是实现低延迟的关键。为此提出上下文感知视频流，仅对对话相关区域分配高码率，显著降低整体带宽。为评估视频质量对MLLM的影响，构建首个基准测试DeViBench。代码已开源。
原文摘要 · Abstract (English)
AI Video Chat emerges as a new paradigm for Real-time Communication (RTC), where one peer is not a human, but a Multimodal Large Language Model (MLLM). This makes interaction between humans and AI more intuitive, as if chatting face-to-face with a real person. However, this poses significant challenges to latency, because the MLLM inference takes up most of the response time, leaving very little time for video streaming. Due to network uncertainty, transmission latency becomes a critical bottleneck preventing AI from being like a real person. To address this, we call for AI-oriented RTC research, exploring the network requirement shift from "humans watching video" to "AI understanding video". We begin by recognizing the main differences between AI Video Chat and traditional RTC. Then, through prototype measurements, we identify that ultra-low bitrate is a key factor for low latency. To reduce bitrate dramatically while maintaining MLLM accuracy, we propose Context-Aware Video Streaming that recognizes the importance of each video region for chat and allocates bitrate almost exclusively to chat-important regions. To evaluate the impact of video streaming quality on MLLM accuracy, we build the first benchmark, named Degraded Video Understanding Benchmark (DeViBench). Finally, we discuss some open questions and ongoing solutions for AI Video Chat. DeViBench is open-sourced at: https://github.com/pku-netvideo/DeViBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。