UniWhisper统一训练多任务音频模型，提升跨领域表现。
UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation
- 将异构音频任务统一为指令-回答格式，用通用语言模型训练。
- 在20项任务上，MLP探针得分0.81，kNN得分0.61，优于Whisper。
- 适合需要统一音频表示的多模态系统开发者使用。
通用音频表示应在一个编码器中捕捉语音的细粒度线索以及环境声和音乐的高层语义。现有编码器通常在某一领域表现优异，但在其他领域性能下降。我们提出UniWhisper，一种高效的持续多任务训练框架，将异构音频任务转化为统一的指令与答案格式，使标准的下一个词预测训练无需特定任务头和损失函数。我们在38,000小时公开音频数据上训练该模型，并通过浅层MLP探测器和k近邻（kNN）在20个任务上评估编码器性能，涵盖语音、环境声和音乐。UniWhisper在MLP探测器下达到0.81的归一化加权平均分，在kNN下达0.61，显著优于Whisper的0.64和0.46，同时保持了强大的语音处理能力。
原文摘要 · Abstract (English)
A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。