arXiv:2608.10659cs.SD2026-08

将视觉自蒸馏方法DINO迁移至音频,实现通用音频表征学习。

DINO-A: Adapting Self-Distillation Vision Transformers to General Audio Representation Learning

  • 用对数梅尔谱图和BYOL-A v2增强策略改造DINO,适配音频任务。
  • 小分辨率块在所有任务上表现更优,视觉变压器优于卷积网络处理环境音与音乐。
  • 在相同条件下,DINO-A比BYOL-A v2低11.96个百分点,源于高维投影与多裁剪开销。

我们提出DINO-A,将自蒸馏方法从视觉领域迁移至通用音频表征学习。尽管DINO已成为自监督视觉的基准方法,而先前音频研究探索了潜在预测(BYOL-A)和掩码建模(Audio-MAE、BEATs),但尚无工作将经典DINO应用于通用音频分类。DINO-A保留DINO的多裁剪、EMA教师模型和高维投影机制,仅将输入模态和增强策略替换为对数梅尔谱图及BYOL-A v2增强块。我们在FSD50K上预训练三种主干网络:两个8x8与16x16块的视觉变换器,以及一个卷积编码器,并在ESC-50、Speech Commands v2、UrbanSound8K和GTZAN上通过线性探测评估。结果表明:视觉变换器内更小的块分辨率始终提升表示质量;卷积网络在语音任务中占优,而视觉变换器在环境音与音乐任务中表现更佳。在相同预训练与评估条件下,DINO-A平均比BYOL-A v2低11.96个百分点,其差异可归因于两点:高维投影空间在FSD50K数据规模下成为负担,以及多裁剪增强带来的额外计算成本——这是DINO-A特有而BYOL-A v2未采用的机制。

原文摘要 · Abstract (English)

We present DINO-A, an adaptation of self-distillation from vision to general audio representation learning. While DINO has become a canonical method in self-supervised vision and prior audio work has explored latent prediction (BYOL-A) and masked modeling (Audio-MAE, BEATs), no prior work has brought canonical DINO to general audio classification in the way BYOL-A brought BYOL. DINO-A retains DINO's multi-crop, EMA teacher, and high-dimensional projection, replacing only the input modality and augmentations with log-mel spectrograms and the BYOL-A v2 augmentation block. We pretrain three backbones, two Vision Transformers with 8x8 and 16x16 patches and a convolutional encoder, on FSD50K and evaluate them with linear probing on ESC-50, Speech Commands v2, UrbanSound8K, and GTZAN. Three findings characterize the resulting representations. Patch resolution within the Vision Transformer family has consistent effect on representation quality, with smaller patches winning across all four tasks. The choice between Vision Transformer and convolutional backbone interacts with task type: convolutional networks lead on speech while Vision Transformers lead on environmental sounds and music. Under identical pretraining and evaluation conditions, DINO-A and BYOL-A v2 differ by 11.96 percentage points on average, and we trace this difference to two mechanisms: the interaction between DINO's high-dimensional projection space and FSD50K's limited scale, and the additional cost of multi-crop augmentation, which DINO uses but BYOL-A v2 does not. The high-dimensional projection space, central to DINO's success in vision, becomes a liability at FSD50K scale.

音频表征自蒸馏视觉变换器对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。