用二维编码器压缩音频，单码本实现高保真还原。
MelTok: 2D Tokenization for Single-Codebook Audio Compression
- 将音频转为二维梅尔谱图，用单码本高效编码
- 在44.1kHz音频上达到与多码本相当的还原质量
- 适合需要精细声学表征的语音理解任务
大型音频语言模型（LALMs）在多种音频理解任务中表现优异，可通过神经音频编解码器进一步提升。将多层残差向量量化器改为单层量化器，有助于提升下游语言模型的解码效率。然而，单码本难以捕捉精细声学细节，因一维编码器的频率变异性导致冗余。为此，我们提出MelTok，一种二维（2D）编码器，能将44.1 KHz音频的声学细节有效压缩至单码本。该编码器生成的表示比一维编码器更紧凑。此外，我们设计了一种基于令牌的声码器，用于从梅尔频谱图令牌恢复音频。客观与主观评估表明，MelTok在高保真音频重建上达到与多码本编解码器相当的水平，并优于现有单码本神经编解码器。通过保留声学细节，MelTok为下游理解任务提供了强表征能力。
原文摘要 · Abstract (English)
Large Audio Language Models (LALMs) have emerged with strong performance across diverse audio understanding tasks and can be further enhanced by neural audio codecs. Transitioning from multi-layer residual vector quantizers to a single-layer quantizer has been shown to facilitate more efficient downstream language models decoding. However, the ability of a single codebook to capture fine-grained acoustic details remains limited, as the frequency-variant nature of 1D tokenizers leads to redundancy. To address this issue, we propose MelTok, a two-dimensional (2D) tokenizer that effectively compresses acoustic details of 44.1 KHz audio into a single codebook. The tokenizer encodes audio into a more compact representation than one-dimensional tokenizers. Furthermore, to recover audio from mel-spectrogram tokens, we propose a token-based vocoder. Both objective and subjective evaluations demonstrate that MelTok achieves quality comparable to multi-codebook codecs and outperforms existing state-of-the-art neural codecs with a single codebook on high-fidelity audio reconstruction. By preserving acoustic details, MelTok offers a strong representation for downstream understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。