用混合音效微调Whisper，提升音乐和环境音识别效果。
Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training
- 在2000万样本上微调Whisper-large-v3，融合语音、音乐与环境音
- 在ESC-50、GTZAN等数据集上性能提升最高达23%
- 可直接替换原模型，降低后续音频大模型训练成本
以语音为主的音频大语言模型通常采用Whisper作为音频编码器，但Whisper仅在语音数据上训练，对音乐和环境音的表征能力较弱，导致下游模型需大量非语音数据进行补偿。本文提出Whisper-AuT，通过在约2000万样本的混合数据（语音80%、环境音10%、音乐10%）上微调Whisper-large-v3获得域适应的音频编码器。整个编码器-解码器结构采用序列到序列的描述生成目标端到端训练，之后丢弃解码器仅保留编码器。线性探测评估显示，Whisper-AuT在ESC-50（环境音分类）上提升23.0%，在GTZAN（音乐流派）上提升5.0%，在Speech Commands（关键词检测）上提升0.7%。该模型设计为Whisper在音频大模型架构中的即插即用替代品，旨在通过提供更强的非语音领域初始表征，降低下游训练成本。
原文摘要 · Abstract (English)
Audio-native large language models (audio-LLMs) commonly use Whisper as their audio encoder. However, Whisper was trained exclusively on speech data, producing weak representations for music and environmental sound. This forces downstream audio-LLMs to compensate through extensive training on large-scale non-speech data. We present Whisper-AuT, a domain-adapted audio encoder obtained by fine-tuning Whisper-large-v3 on a curated mixture of speech (80%), environmental sound (10%), and music (10%) totaling approximately 20M samples. The full encoder-decoder is trained end-to-end with a seq2seq captioning objective; the decoder is then discarded and only the encoder is retained. Linear probe evaluations show that Whisper-AuT achieves +23.0% on ESC-50 (environmental sound), +5.0% on GTZAN (music genre), and +0.7% on Speech Commands (keyword spotting) compared to the original Whisperlarge-v3 encoder. Whisper-AuT is designed as a drop-in replacement for Whisper in audio-LLM architectures, with the goal of reducing downstream training cost by providing stronger initial audio representations for non-speech domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。