一个90亿参数模型,能听懂音频并生成语音,还能精准编辑。
FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

- 用不同路径处理理解与生成,分离连续表征提升效果
- 支持长达一小时音频理解,零样本语音合成准确率高
- 适合语音生成、编辑与长时音频分析的研究者使用
统一音频模型需同时识别语言、语调和环境信息,并支持语音合成与编辑。核心挑战在于表征:理解需要紧凑特征以适配长上下文建模,而生成则需可重构特征以保留细微声学细节。我们提出FireRedAudio,一个具有共享90亿参数大语言模型的通用音频语言模型。据我们所知,它是首个公开的统一音频-语言模型,在单个可训练自回归大模型中实现了理解与生成的分离连续输入表征。待识别或分析的音频由专用音频编码器处理,生成用的语音输入通过RedAE基路径处理。该大语言模型直接生成文本,或条件化流匹配扩散模型(DiT)生成连续声学隐变量。通过渐进多任务训练,FireRedAudio支持语音识别与音频理解(扩展至长达一小时的录音),以及零样本语音合成、指令语音合成、语义与声学语音编辑。其对长时音频的结构化组织实现秒级时间戳精度。在全面评估中,模型在音频理解与多语言语音识别上表现竞争或领先;零样本语音合成具备强内容准确性与说话人保真度;指令语音合成表现领先;在语义与声学语音编辑方面,显著优于Ming-UniAudio-Edit。这些结果证明了解耦连续输入表征在中等规模模型中统一音频理解与连续潜变量语音生成的可行性。代码已开源:https://github.com/FireRedTeam/FireRedAudio。
原文摘要 · Abstract (English)
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。