开源音频大模型，用通用音效描述实现高效全面的音频理解
MiDashengLM: Efficient Audio Understanding with General Audio Captions
- 基于公开数据训练，用通用音效描述融合语音、声音与音乐
- 推理速度提升4倍，吞吐量最高达同类模型20倍
- 适合需要透明可复现音频理解方案的研究者
当前大型音频语言模型（LALMs）多依赖封闭数据源或专有模型，限制了泛化能力与可访问性。本文提出MiDashengLM，一种新型开源音频语言模型，通过作者自建的ACAVCaps训练数据集，利用通用音频描述实现高效且全面的音频理解。该模型仅使用公开可用的预训练和监督微调（SFT）数据集，确保完全透明与可复现。核心采用开源音频编码器Dasheng，专门设计用于有效处理多样化听觉信息。不同于以往以语音识别（ASR）为基础的音文对齐方法，本工作聚焦于通用音频描述，将语音、声音与音乐信息融合为统一文本表征，实现复杂音频场景的完整语义表达。最终，MiDashengLM在时间到首个令牌（TTFT）上实现最高4倍加速，在吞吐量上较同类模型提升高达20倍。模型检查点已发布于Hugging Face与GitHub。
原文摘要 · Abstract (English)
Current approaches for large audio language models (LALMs) often rely on closed data sources or proprietary models, limiting their generalization and accessibility. This paper introduces MiDashengLM, a novel open audio-language model designed for efficient and comprehensive audio understanding through the use of general audio captions using our novel ACAVCaps training dataset. MiDashengLM exclusively relies on publicly available pretraining and supervised fine-tuning (SFT) datasets, ensuring full transparency and reproducibility. At its core, MiDashengLM integrates Dasheng, an open-source audio encoder, specifically engineered to process diverse auditory information effectively. Unlike previous works primarily focused on Automatic Speech Recognition (ASR) based audio-text alignment, our strategy centers on general audio captions, fusing speech, sound and music information into one textual representation, enabling a holistic textual representation of complex audio scenes. Lastly, MiDashengLM provides an up to 4x speedup in terms of time-to-first-token (TTFT) and up to 20x higher throughput than comparable models. Checkpoints are available online at https://huggingface.co/mispeech/midashenglm-7b and https://github.com/xiaomi-research/dasheng-lm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。