提出高效稳定的音频建模架构AudioRWKV,突破长音频处理瓶颈。
AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition
- 用2D深度可分离卷积替代原token-shift,更好捕捉声谱时空特征
- 引入双向WKV核,在线性复杂度下实现全局上下文建模
- 模型稳定易扩展,长音频处理速度提升13.3倍,性能媲美大模型
近期,Transformer(如Audio Spectrogram Transformers, AST)和状态空间模型(如Audio Mamba, AuM)在音频建模中取得显著进展。然而,Transformer的O(L²)计算复杂度限制了长序列高效处理,而Mamba在参数和数据规模增大时易出现不稳定问题。为此,本文提出AudioRWKV(A-RWKV),一种高效且稳定的音频建模架构。具体地,继承RWKV7的稳定递归结构,将原1D token-shift操作替换为2D深度可分离卷积,以更好地捕获局部声谱-时序模式;同时,将原始因果WKV核改进为双向WKV核(Bi-WKV),在保持线性计算复杂度的前提下实现整个音频序列的全局上下文建模。得益于RWKV7固有的稳定性,A-RWKV可无缝扩展至更大模型规模。实验表明,在相同线性模型设置下,A-RWKV-S(22M)性能与AuM-B(92M)相当,且吞吐更稳定;对于长音频(约5分钟28秒),WKV7处理速度最高提升13.3倍。
原文摘要 · Abstract (English)
Recently, Transformers (e.g., Audio Spectrogram Transformers, AST) and state-space models (e.g., Audio Mamba, AuM) have achieved remarkable progress in audio modeling. However, the O(L^2) computational complexity of the Transformer architecture hinders efficient long-sequence processing, while the Mamba architecture tends to become unstable when scaling parameters and data. To address these challenges, this paper proposes AudioRWKV (A-RWKV), a highly efficient and stable architecture for audio modeling. Specifically, we inherit the stable and efficient recurrent formulation of RWKV7 and replace its 1D token-shift operation with a 2D depthwise separable convolution to better capture local spectro-temporal patterns. Furthermore, we adapt the original causal WKV kernel into a bidirectional WKV kernel (Bi-WKV), enabling global context modeling over the entire audio sequence while maintaining linear computational complexity. Benefiting from the inherent stability of the RWKV7 foundation, A-RWKV scales seamlessly to larger model sizes. Experimental results demonstrate that, under the same linear-model regime, A-RWKV-S (22M) achieves performance parity with AuM-B (92M) while exhibiting more stable throughput than AST; for long-form audio (~5 minutes 28 seconds), WKV7 achieves up to a 13.3X speedup in processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。