实现音视频实时生成,速度达每秒25帧且同步精准
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
- 将双向扩散模型蒸馏为流式自回归生成框架
- 在单卡上实现约25帧/秒的实时生成,保持高质量同步
- 适合需要低延迟音视频生成的应用场景
近期的音视频联合扩散模型虽生成质量高,但因双向注意力依赖导致延迟过高,难以支持实时应用。本文提出OmniForcing,首个将离线双流双向扩散模型蒸馏为高保真流式自回归生成器的框架。直接应用因果蒸馏会因模态间严重的时间不对称和标记稀疏性引发训练不稳定。为此,我们引入非对称块因果对齐与零截断全局前缀,防止多模态同步漂移;通过配备身份RoPE约束的音频掩码令牌机制,解决因果转换中音频标记极度稀疏导致的梯度爆炸问题;最终采用联合自强迫蒸馏范式,在长序列推理中动态修正暴露偏差带来的累积跨模态误差。依托与模态无关的滚动键值缓存推理方案,OmniForcing在单张GPU上实现约25帧/秒的领先流式生成性能,同时保持与双向教师模型相当的多模态同步性与视觉质量。
原文摘要 · Abstract (English)
Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。