提出新型音频分块方法，提升分类准确率并大幅降低计算量
Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification
- 按时间局部性跨全频带分块，保留谐波结构
- 在AudioSet和SpeechCommands上分别提升6.76%和8.46%准确率
- 适配Transformer与状态空间模型，适合追求效率的音频任务
Transformers与状态空间模型（SSMs）通过将频谱图视为补丁序列推动了音频分类的发展。然而，现有模型如音频频谱变换器（AST）和音频马比（AuM）采用来自计算机视觉的方形分块方式，破坏了连续频率模式，导致补丁数量过多，训练变慢且计算开销大。本文提出全频带时序分块（FFTP），通过在局部时间上下文中跨越全频带进行分块，更好地匹配频谱图的时间-频率不对称性，保留谐波结构，显著减少补丁数量与计算量。同时引入SpecMask，一种补丁对齐的频谱增强方法，在固定掩码预算下结合全频带与局部时空掩码，提升时间鲁棒性的同时保持频谱连续性。在AST与AuM上应用该分块方法与SpecMask后，音频分类在AudioSet-18k上mAP提升最高达+6.76，在SpeechCommandsV2上准确率提升最高达+8.46，同时计算量减少最高达83.26%，验证了性能与效率的双重优势。
原文摘要 · Abstract (English)
Transformers and State-Space Models (SSMs) have advanced audio classification by modeling spectrograms as sequences of patches. However, existing models such as the Audio Spectrogram Transformer (AST) and Audio Mamba (AuM) adopt square patching from computer vision, which disrupts continuous frequency patterns and produces an excessive number of patches, slowing training, and increasing computation. We propose Full-Frequency Temporal Patching (FFTP), a patching strategy that better matches the time-frequency asymmetry of spectrograms by spanning full frequency bands with localized temporal context, preserving harmonic structure, and significantly reducing patch count and computation. We also introduce SpecMask, a patch-aligned spectrogram augmentation that combines full-frequency and localized time-frequency masks under a fixed masking budget, enhancing temporal robustness while preserving spectral continuity. When applied on both AST and AuM, our patching method with SpecMask improves mAP by up to +6.76 on AudioSet-18k and accuracy by up to +8.46 on SpeechCommandsV2, while reducing computation by up to 83.26%, demonstrating both performance and efficiency gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。