实时生成视频对应音频,支持直播和交互应用
SoundReactor: Frame-level Online Video-to-Audio Generation
- 采用因果Transformer架构,逐帧生成音频不依赖未来视频帧
- 单帧延迟仅26.3毫秒(每秒30帧,480p),可实现实时响应
- 首次实现端到端在线音视频同步,适合游戏直播等场景
现有视频转音频(V2A)模型多为离线处理,需完整视频序列或分块输入,难以用于直播创作与生成式世界模型等交互场景。为此,我们提出帧级在线V2A生成新任务,即模型在无未来视频帧前提下,自回归生成音频。我们提出SoundReactor,据知是首个专为此任务设计的简洁高效框架。其基于仅解码器的因果Transformer,以连续音频隐变量为输出;视觉条件通过DINOv2最小变体提取的网格特征生成每帧单个标记,保持端到端因果性与效率。模型经扩散预训练后进行一致性微调,加速扩散头解码。在来自顶级游戏大作的多样化玩法视频数据集上,模型成功生成语义与时间对齐、高保真全频段立体声音频,经客观与人工评估验证。单张H100显卡下,30FPS、480p视频的单帧波形级延迟为26.3ms(NFE=1)、31.5ms(NFE=4)。
原文摘要 · Abstract (English)
Prevailing Video-to-Audio (V2A) generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand. This critically limits their use in interactive applications such as live content creation and emerging generative world models. To address this gap, we introduce the novel task of frame-level online V2A generation, where a model autoregressively generates audio from video without access to future video frames. Furthermore, we propose SoundReactor, which, to the best of our knowledge, is the first simple yet effective framework explicitly tailored for this task. Our design enforces end-to-end causality and targets low per-frame latency with audio-visual synchronization. Our model's backbone is a decoder-only causal transformer over continuous audio latents. For vision conditioning, it leverages grid (patch) features extracted from the smallest variant of the DINOv2 vision encoder, which are aggregated into a single token per frame to maintain end-to-end causality and efficiency. The model is trained through a diffusion pre-training followed by consistency fine-tuning to accelerate the diffusion head decoding. On a benchmark of diverse gameplay videos from AAA titles, our model successfully generates semantically and temporally aligned, high-quality full-band stereo audio, validated by both objective and human evaluations. Furthermore, our model achieves low per-frame waveform-level latency (26.3ms with the head NFE=1, 31.5ms with NFE=4) on 30FPS, 480p videos using a single H100. Demo samples are available at https://koichi-saito-sony.github.io/soundreactor/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。