只需4步采样,就能高效生成高质量音频,支持文本、视频、音频多种控制信号。
AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation

- 采用教师-学生框架,通过分布匹配蒸馏实现快速生成。
- 在920万样本数据集上训练,4步采样性能超越多步基线。
- 适合需要快速响应的音频生成场景,如交互式创作或实时应用。
基于灵活多模态控制信号的音频与音乐生成是广泛应用领域,面临三大挑战:统一多模态建模框架、大规模高质量训练数据、以及多步扩散采样的高昂推理成本。为此,本文提出 AudioX-Turbo,一种统一且高效的任意输入到音频生成框架,可整合文本、视频和音频等多种条件信号。AudioX-Turbo 采用教师-学生范式:教师模型 AudioX-Base 基于多模态扩散变换器与自适应融合模块,实现多源输入对齐以生成高保真音频;随后通过适配流匹配的分布匹配蒸馏方法,将其压缩为仅需4步采样的学生模型。同时引入基于扩散的判别器,保障少步生成质量。为支持训练,构建了包含约920万样本的大规模高质量数据集 IF-caps-Pro,通过两阶段数据收集与标注流程获得。在多种任务上进行评估,结果表明该模型在文本到音频和文本到音乐生成中表现优异,仅需4次采样步骤,函数求值次数(NFE)比多步基线减少约25倍,验证了其在多模态控制下的高效与强指令遵循能力。代码与数据集将公开于 https://zeyuet.github.io/AudioX-Turbo/。
原文摘要 · Abstract (English)
Audio and music generation based on flexible multimodal control signals is a widely applicable topic, with the following key challenges: 1) a unified multimodal modeling framework, 2) large-scale, high-quality training data, and 3) the prohibitive inference cost of multi-step diffusion sampling. As such, we propose AudioX-Turbo, a unified and efficient framework for anything-to-audio generation that integrates varied multimodal conditions (i.e., text, video, and audio signals) in this work. AudioX-Turbo follows a teacher-student paradigm. The teacher AudioX-Base is built on a Multimodal Diffusion Transformer with a Multimodal Adaptive Fusion module that aligns diverse multimodal inputs for high-fidelity synthesis, and is then distilled into the few-step student AudioX-Turbo via Distribution Matching Distillation adapted to flow matching, complemented by a diffusion-based discriminator for high-quality few-step generation. To support the training of AudioX-Turbo, we construct a large-scale, high-quality dataset, IF-caps-Pro, comprising approximately 9.2M samples curated through a two-stage data collection and annotation pipeline. We benchmark AudioX-Turbo across a wide range of tasks, finding that our model achieves superior performance, especially on text-to-audio and text-to-music generation, while operating at only 4 sampling steps and requiring approximately 25x fewer function evaluations (NFE) than multi-step baselines. These results demonstrate that our method is capable of audio generation under flexible multimodal control, showing efficient and powerful instruction-following capabilities. The code and datasets will be available at https://zeyuet.github.io/AudioX-Turbo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。