发现音频离散化存在不一致问题，提出有效缓解方法。
Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models
- 分析神经音频编码器中离散音频标记的上下文依赖性差异
- 在LibriTTS和MLS数据集上减少生成中的重复与遗漏
- 适合语音生成、音频建模研究者参考
受大型语言模型启发，音频生成任务越来越多地采用离散音频标记序列。然而，直接通过神经音频编码器离散化音频会导致序列与文本序列本质不同：同一段音频在不同上下文中可能生成多个不同标记序列，尽管听感完全相同。我们称此现象为离散表示不一致性（DRI）。该问题导致同一音频被多个不同序列表示，造成神经编码语言模型生成时出现重复与遗漏。本文对EnCodec等主流音频标记器中的DRI现象进行定量分析，并提出有效缓解方法。在LibriTTS及大规模MLS数据集（44,000小时）上的实验表明，该方法在提升生成质量与一致性方面具有显著效果与通用性。演示音频样本可在线查看。
原文摘要 · Abstract (English)
Building upon advancements in Large Language Models (LLMs), the field of audio processing has seen increased interest in training audio generation tasks with discrete audio token sequences. However, directly discretizing audio by neural audio codecs often results in sequences that fundamentally differ from text sequences. Unlike text, where text token sequences are deterministic, discrete audio tokens can exhibit significant variability based on contextual factors, while still producing perceptually identical audio segments. We refer to this phenomenon as \textbf{Discrete Representation Inconsistency (DRI)}. This inconsistency can lead to a single audio segment being represented by multiple divergent sequences, which creates confusion in neural codec language models and results in omissions and repetitions during speech generation. In this paper, we quantitatively analyze the DRI phenomenon within popular audio tokenizers such as EnCodec. Our approach effectively mitigates the DRI phenomenon of the neural audio codec. Furthermore, extensive experiments on the neural codec language model over LibriTTS and large-scale MLS datases (44,000 hours) demonstrate the effectiveness and generality of our method. The demo of audio samples is available online~\footnote{\url{https://consistencyinneuralcodec.github.io}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。