用Transformer模型生成虎鲸叫声，可还原真实声学特征。
WhAM: Towards A Translative Model of Sperm Whale Vocalization
- 基于预训练音乐模型微调，通过掩码预测生成叫声
- 合成叫声与真实录音的音高、节奏匹配度高
- 适合海洋生物学家研究鲸类交流行为
虎鲸通过短序列点击声（称为回声模式）进行交流。我们提出WhAM（鲸类声学模型），首个基于Transformer的模型，能从任意音频提示生成合成虎鲸回声模式。WhAM通过在二十年间收集的1万段回声模式数据上微调VampNet（一个预训练于音乐音频的掩码声学词元模型）构建。通过迭代掩码词元预测，WhAM生成的合成回声模式保留了原始录音的关键声学特征。我们使用弗雷切特音频距离和专家海洋生物学家的感知实验评估合成效果。在节奏、社会群体和元音分类等下游任务中，尽管未专门针对分类训练，WhAM学习到的表征仍表现优异。代码已开源：https://github.com/Project-CETI/wham
原文摘要 · Abstract (English)
Sperm whales communicate in short sequences of clicks known as codas. We present WhAM (Whale Acoustics Model), the first transformer-based model capable of generating synthetic sperm whale codas from any audio prompt. WhAM is built by finetuning VampNet, a masked acoustic token model pretrained on musical audio, using 10k coda recordings collected over the past two decades. Through iterative masked token prediction, WhAM generates high-fidelity synthetic codas that preserve key acoustic features of the source recordings. We evaluate WhAM's synthetic codas using Fréchet Audio Distance and through perceptual studies with expert marine biologists. On downstream classification tasks including rhythm, social unit, and vowel classification, WhAM's learned representations achieve strong performance, despite being trained for generation rather than classification. Our code is available at https://github.com/Project-CETI/wham
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。