Kimi-Audio是开源音频基础模型，支持语音理解、生成与对话，性能领先。
Kimi-Audio Technical Report
- 采用12.5Hz音频分词器与基于LLM的架构，输入连续特征输出离散令牌。
- 使用超1300万小时多模态音频数据预训练，支持语音识别等任务的顶尖表现。
- 适合研究音频生成、语音交互的开发者，代码和模型已开源。
我们提出Kimi-Audio，一个开源的音频基础模型，在音频理解、生成与对话方面表现卓越。论文详述了模型构建实践，包括架构设计、数据筛选、训练方案、推理部署与评估方法。具体而言，采用12.5Hz音频分词器，设计基于大语言模型的新型架构，以连续特征为输入、离散标记为输出，并开发基于流匹配的分块流式反分词器。预训练数据集包含超过1300万小时音频，涵盖语音、声音与音乐等多种模态，并构建高质量后训练数据流水线。模型从预训练语言模型初始化，持续在音频与文本数据上进行预训练，结合多个精心设计的任务，随后微调以支持多样化的音频任务。大量评估显示，Kimi-Audio在语音识别、音频理解、音频问答与语音对话等多个基准上达到当前最优水平。相关代码、模型检查点与评估工具已在https://github.com/MoonshotAI/Kimi-Audio发布。
原文摘要 · Abstract (English)
We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。