研究预训练音频嵌入对常见音效的敏感性，发现其变形轨迹非线性且高维。
Investigating the Sensitivity of Pre-trained Audio Embeddings to Common Effects
- 用参数化音效分析嵌入空间中的变形轨迹，量化其维度与线性程度。
- 嵌入随音效强度单调移动，但位移子空间普遍为高维，不具全局线性。
- 投影去除估计变形方向无法提升下游分类任务的鲁棒性，适合关注模型脆弱性的研究者。
近年来，基础模型在多个领域显著推进了数据驱动系统的发展。然而，它们作为特征提取器的内在特性仍缺乏深入探索。本文研究了广泛使用的基础模型（OpenL3、PANNs、CLAP）中音频嵌入对常见音效的敏感性。由于音效在大规模音频数据集中普遍存在，我们重点考察其影响。通过施加参数化音效（增益、低通滤波、混响、比特破碎），分析嵌入空间中变形轨迹与音效强度之间的相关性。我们提出使用典型相关分析来量化变形轨迹的维度和线性程度。结果表明，存在一个方向使嵌入随音效强度单调变化，但包含位移的子空间通常为高维，说明预训练音频嵌入并未全局线性化音效影响。下游乐器分类任务的实验证明，投影去除估计的变形方向无法普遍提升嵌入对音效的鲁棒性。
原文摘要 · Abstract (English)
In recent years, foundation models have significantly advanced data-driven systems across various domains. Yet, their underlying properties, especially when functioning as feature extractors, remain under-explored. In this paper, we investigate the sensitivity to audio effects of audio embeddings extracted from widely-used foundation models, including OpenL3, PANNs, and CLAP. We focus on audio effects as the source of sensitivity due to their prevalent presence in large audio datasets. By applying parameterized audio effects (gain, low-pass filtering, reverberation, and bitcrushing), we analyze the correlation between the deformation trajectories and the effect strength in the embedding space. We propose to quantify the dimensionality and linearizability of the deformation trajectories induced by audio effects using canonical correlation analysis. We find that there exists a direction along which the embeddings move monotonically as the audio effect strength increases, but that the subspace containing the displacements is generally high-dimensional. This shows that pre-trained audio embeddings do not globally linearize the effects. Our empirical results on instrument classification downstream tasks confirm that projecting out the estimated deformation directions cannot generally improve the robustness of pre-trained embeddings to audio effects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。