用大模型+流匹配实现端到端音频场景生成,语音更清晰。
MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

- 用预训练大模型驱动逐词条件流匹配,实现端到端生成
- 英文语音识别错误率从12.15%降到2.79%,接近专用语音系统
- 支持多语言生成,且混合音频质量表现优秀
生成同时包含语音、音乐和音效的连贯音频场景仍具挑战。现有方法通常采用分离式流程,使用固定文本编码器配合独立音频解码器,限制了跨模态优化,导致语音可懂度差。为此,我们提出MiDashengLM-Gen,一种将预训练大语言模型(LLM)与逐标记条件流匹配结合的端到端框架,实现自回归、变长混合音频场景生成。该框架是首个统一文本到音频生成的端到端训练模型。实证评估显示,其显著提升现有统一模型的语音可懂度:在Seed-TTS基准上,英文词错误率(WER)从12.15%降至2.79%,接近专用语音合成系统(1.24%)。此外,该框架在多语言设置下表现优异,达到有竞争力的多语言WER。最后,在MECAT基准上,混合音频生成质量保持领先。代码与模型权重见https://github.com/xiaomi-research/midashenglm-gen 和 https://huggingface.co/mispeech/midashenglm-gen,演示页为 https://xingws.github.io/midashenglm-gen-demo/。
原文摘要 · Abstract (English)
Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。