arXiv:2607.26818cs.CV2026-07

Ripple实现低延迟音视频实时生成,支持长时序内容连贯输出。

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

论文配图:Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
图 1 · 摘自论文原文
  • 用跨模态循环记忆+滑窗注意力,高效保留长期上下文。
  • 480P下达28帧/秒,比教师模型更快且支持长视频生成。
  • 适合需要实时音视频同步的应用,如直播、虚拟主播。

音视频生成模型虽质量高,但延迟大,难以用于实时场景。现有流式方法成本高且不支持长时生成。为此,我们提出Ripple,一种基于跨模态循环记忆的实时联合音视频生成系统。通过固定长度滑窗注意力与模态专属记忆状态,持续总结音视频上下文,并引入跨模态记忆交互增强音视频同步性。训练采用三阶段方案:(1) 将双向教师模型转为带模拟记忆的块级因果注意力;(2) 通过端到端蒸馏优化记忆构建与交互;(3) 针对流式生成设计在线强化后训练。实验表明,Ripple在480P下实现约28 FPS,速度超过教师模型,且可生成连贯长视频。在短视频与长视频基准上均优于现有离线与在线方法。

原文摘要 · Abstract (English)

Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.

音视频生成实时生成跨模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。