WavFlow直接在波形空间生成高质量音频，无需压缩中间表示。
WavFlow: Audio Generation in Waveform Space
- 将音频转为2D令牌网格并提升幅度，实现波形空间的稳定流匹配生成
- 在VGGSound和AudioCaps上性能媲美甚至超越基于潜空间的方法
- 适合追求简洁高效、多模态音频生成的研究者与开发者
当前音频生成主要依赖潜空间压缩，引入额外复杂性与信息损失。本文提出WavFlow，一种直接在原始波形空间生成高保真音频的框架。为克服高维低能信号建模难题，通过波形分块重构音频为2D令牌网格，并引入幅度提升以对齐信号尺度，实现通过流匹配的直接x-预测稳定优化。为捕捉复杂的语义对齐与时间同步，利用自动化数据管道构建了包含500万条高质量视频-文本-音频三元组的数据集，使模型从零开始学习精细声学模式。实验表明，WavFlow在视频到音频基准VGGSound（FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44）和文本到音频基准AudioCaps（FD_PANNs: 10.63, IS_PANNs: 12.62）上表现优异，达到或超过现有潜空间方法水平。研究证明，中间压缩并非高质量合成的必要条件，为多模态音频生成提供更简洁可扩展的替代方案。
原文摘要 · Abstract (English)
Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。