一个神经音频编解码器可动态调整音频令牌时间分辨率。
Neural Audio Codec with Adjustable Token Temporal Resolution Using Sampling-Frequency-Independent Convolutional Layers
- 用与采样率无关的卷积层生成适配不同时间分辨率的核。
- 同一模型在多种时间分辨率下表现优于独立训练的单模型。
- 适合需要灵活处理音频时序的生成与理解任务。
从神经音频编解码器（NAC）获得的离散令牌被用作音频生成和理解模型中的紧凑表示。在基于令牌的系统中，令牌时间分辨率（TTR）——即相邻令牌帧之间的时间间隔——至关重要，因为它控制着快速声学事件的表示能力与令牌序列长度之间的权衡。然而，大多数NAC仅在一个TTR下训练，需为每个TTR单独训练。本文提出一种机制，使单一NAC可在多个TTR下运行，该机制利用与采样率无关的卷积层，将TTR视为令牌序列的采样周期，从共享参数集中生成依赖TTR的卷积核，并针对每种TTR调整核大小与步幅。我们将该机制集成到Descript Audio Codec中，保持量化器不变。环境声音重建实验表明，所提模型在性能上优于使用特定于TTR层切换的单模型基线。
原文摘要 · Abstract (English)
Discrete tokens obtained from neural audio codecs (NACs) have been used as compact representations in audio generation and understanding models. In such token-based systems, token temporal resolution (TTR), defined as the time interval between adjacent token frames, is important because it controls the trade-off between representing rapid acoustic events and reducing token-sequence length. However, most NACs are trained at a single TTR and require separate training for each TTR. This paper proposes a mechanism that enables a single NAC to operate at multiple TTRs using sampling-frequency-independent convolutional layers. The mechanism regards TTR as the sampling period of the token sequence and generates TTR-dependent convolutional kernels from a shared parameter set, while adjusting the kernel size and stride for each TTR. We incorporate the mechanism into Descript Audio Codec, leaving the quantizer unchanged. Experiments on environmental sound reconstruction show that the proposed model outperforms a single-model baseline that switches TTR-specific layers for each TTR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。