通过精简采样步数和每步计算量，实现音乐生成速度10-18倍提升。
Presto! Distilling Steps and Layers for Accelerating Music Generation
- 采用基于GAN的分布匹配蒸馏法减少采样步数。
- 改进隐藏状态方差保留，降低每步计算成本。
- 结合双策略，生成质量高且速度快，适合实时音乐生成场景。
尽管基于扩散模型的文本到音乐生成方法取得进展，但高效且高质量的生成仍具挑战。本文提出Presto!，一种通过减少采样步数和每步计算成本来加速基于分数的扩散变压器的方法。为减少步数，我们开发了一种新型基于分数的分布匹配蒸馏（DMD）方法，首次将GAN用于文本到音乐生成的蒸馏；为降低每步开销，改进了近期层蒸馏方法，通过更好保留隐藏状态方差提升学习效果。二者结合形成双重优化策略。独立评估显示，两种蒸馏方法均达当前最优性能。联合使用时，可在32秒单声道/立体声44.1kHz音乐生成中实现230/435毫秒延迟，比现有最佳方法快15倍，是目前已知最快的高质量文本到音乐生成系统。音频示例见https://presto-music.github.io/web/。
原文摘要 · Abstract (English)
Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge. Sound examples can be found at https://presto-music.github.io/web/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。