音频预训练模型不微调效果反而更差，需谨慎用于生物声学分析。
No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings
- 对比11种模型，未微调的预训练模型性能低于微调的AlexNet。
- 无论是否微调，多数模型无法区分背景音与目标声音，唯独ResNet能做到。
- 微调时减少背景音数量，模型表现显著提升，提示数据选择关键。
生物声学通过动物声音非侵入式监测生态系统。当前流行直接使用音频预训练深度学习模型提取嵌入向量，无需微调即可获取特征。然而，本基准研究发现，尽管微调后的VGG和变换器模型在某些任务上表现领先，但在其他任务中却表现不佳。该研究在相同任务上对11种深度学习模型进行评估，通过降维嵌入并进行聚类分析。结果表明：1）未微调的音频预训练模型性能甚至低于微调的AlexNet；2）无论是否微调，多数模型均无法将背景噪声与标注声音分离，仅ResNet能有效区分；3）在微调过程中若减少背景声音样本，模型性能显著提升。研究强调必须对音频预训练模型进行微调，并在微调后验证嵌入质量。代码已开源：https://github.com/NeuroscienceAI/Audio_Embeddings。
原文摘要 · Abstract (English)
Bioacoustics, the study of animal sounds, offers a non-invasive method to monitor ecosystems. Extracting embeddings from audio-pretrained deep learning (DL) models without fine-tuning has become popular for obtaining bioacoustic features for tasks. However, a recent benchmark study reveals that while fine-tuned audio-pretrained VGG and transformer models achieve state-of-the-art performance in some tasks, they fail in others. This study benchmarks 11 DL models on the same tasks by reducing their learned embeddings' dimensionality and evaluating them through clustering. We found that audio-pretrained DL models 1) without fine-tuning even underperform fine-tuned AlexNet, 2) both with and without fine-tuning fail to separate the background from labeled sounds, but ResNet does, and 3) outperform other models when fewer background sounds are included during fine-tuning. This study underscores the necessity of fine-tuning audio-pretrained models and checking the embeddings after fine-tuning. Our codes are available: https://github.com/NeuroscienceAI/Audio\_Embeddings
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。