解决语音对话模型的效率与记忆难题，实现高质量语音理解与生成。
Fun-Audio-Chat Technical Report
- 采用双分辨率语音表示，5Hz处理保效率，25Hz生成保质量。
- 多阶段训练避免文本大模型知识遗忘，提升语音理解与共情能力。
- 开源8B与MoE模型，支持全双工语音交互，适合语音应用研发者。
近期联合语音-文本模型在无缝语音交互方面展现出巨大潜力，但现有模型面临语音标记（25Hz）与文本标记（约3Hz）时间分辨率不匹配的问题，导致语义信息稀释、计算成本高，并引发文本大模型知识的灾难性遗忘。本文提出Fun-Audio-Chat，一种大音频语言模型，通过两项源自先前工作DrVoice的创新克服上述局限。第一，双分辨率语音表征（DRSR）：共享语言模型以高效5Hz处理音频（通过标记分组），语音精调头则生成25Hz高质量标记，兼顾效率（约节省50% GPU）与质量。第二，核心鸡尾酒训练（Core-Cocktail Training），一种两阶段微调并含中间合并机制，缓解灾难性遗忘。随后采用多任务DPO训练增强鲁棒性、语音理解、指令遵循与语音共情能力。该多阶段后训练使Fun-Audio-Chat在保留文本大模型知识的同时，具备强大的语音理解、推理与生成能力。不同于需大规模音视频预训练的近期大音频语言模型，Fun-Audio-Chat仅依赖预训练模型与大量后训练。8B与MoE 30B-A3B模型在语音转文本、语音转语音任务上表现优异，在同类规模模型中于口语问答基准上排名领先。同时在音频理解、语音功能调用、指令遵循与语音共情任务上达到竞争或更优性能。我们开发了全双工变体Fun-Audio-Chat-Duplex，其在口语问答与全双工交互中表现强劲。项目已开源8B模型及训练推理代码，并提供互动演示，地址为https://github.com/FunAudioLLM/Fun-Audio-Chat。
原文摘要 · Abstract (English)
Recent advancements in joint speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between speech tokens (25Hz) and text tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding, reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on Speech-to-Text and Speech-to-Speech tasks, ranking top among similar-scale models on Spoken QA benchmarks. They also achieve competitive to superior performance on Audio Understanding, Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo, at https://github.com/FunAudioLLM/Fun-Audio-Chat .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。