用更小模型实现更快更准的视觉引导声音生成
MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
- 用掩码扩散变换器替代传统U-Net，减少参数和内存占用
- 500万参数模型达97.9%准确率，比最先进模型快36倍
- 适合资源受限场景下的高效音频生成应用
我们提出MDSGen，一种面向视觉引导开放域声音生成的新框架，专为降低模型参数量、内存消耗和推理速度而优化。该框架包含两项关键创新：(1) 去除冗余视频特征的模块，过滤不必要的视觉信息；(2) 利用时间上下文提升音频生成精度的时序感知掩码策略。与现有资源密集型U-Net模型不同，MDSGen采用去噪掩码扩散变换器，无需依赖预训练扩散模型即可实现高效生成。在基准数据集VGGSound上，最小模型（500万参数）达到97.9%对齐准确率，参数量减少172倍，内存降低371%，推理速度提升36倍，优于当前8.6亿参数的顶尖模型（93.9%准确率）。更大模型（1.31亿参数）接近99%准确率，参数量仅需其6.5倍。结果表明该方法具有优异的可扩展性与有效性。代码已公开于https://bit.ly/mdsgen。
原文摘要 · Abstract (English)
We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, \texttt{MDSGen} employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves $97.9$% alignment accuracy, using $172\times$ fewer parameters, $371$% less memory, and offering $36\times$ faster inference than the current 860M-parameter state-of-the-art model ($93.9$% accuracy). The larger model (131M parameters) reaches nearly $99$% accuracy while requiring $6.5\times$ fewer parameters. These results highlight the scalability and effectiveness of our approach. The code is available at https://bit.ly/mdsgen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。