用多粒度特征与正则化加速音频理解模型训练
Representation-Regularized Convolutional Audio Transformer for Audio Understanding
- 引入多粒度模块捕捉不同层次的音频特征
- 通过正则化使收敛速度提升5倍，性能更优
- 适合需要高效训练的音频分类任务
基于自监督学习（SSL）的方法在音频理解领域取得了显著进展。然而，现有方法通常仅在单一粒度下操作，难以建模复杂音频信号中多样化的时序与频谱结构。此外，从零开始生成表示计算成本高，常需大量训练才能收敛。本文提出卷积音频变换器（CAT），一个统一框架以应对上述挑战。首先，为捕捉层次化音频特征，CAT引入多粒度块，在不同粒度间聚合信息；其次，为提升训练效率，提出表示正则化目标。该辅助任务借鉴生成建模思想，利用冻结的预训练外部编码器提供的高质量语义表示，引导学生模型预测。实验表明，CAT在多个音频理解基准上显著优于基线方法。特别地，在AudioSet 20k数据集上实现竞争性性能，且收敛速度比现有方法快5倍。代码与检查点将很快发布于https://github.com/realzhouchushu/CAT。
原文摘要 · Abstract (English)
Bootstrap-based Self-Supervised Learning (SSL) has achieved remarkable progress in audio understanding. However, existing methods typically operate at a single level of granularity, limiting their ability to model the diverse temporal and spectral structures inherent in complex audio signals. Furthermore, bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge. In this work, we propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges. First, to capture hierarchical audio features, CAT incorporates a Multi-resolution Block that aggregates information across varying granularities. Second, to enhance training efficiency, we introduce a Representation Regularization objective. Drawing inspiration from generative modeling, this auxiliary task guides the student model by aligning its predictions with high-quality semantic representations from frozen, pre-trained external encoders. Experimental results demonstrate that CAT significantly outperforms baselines on audio understanding benchmarks. Notably, it achieves competitive performance on the AudioSet 20k dataset with 5 times faster convergence than existing methods. Codes and checkpoints will be released soon at https://github.com/realzhouchushu/CAT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。