通过对齐解码器特征流形，提升语音编码器重建质量。
Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment
- 用轻量映射损失对齐解码器处理量化与连续嵌入时的特征流形。
- 在低比特率下实现当前最佳性能，4倍减小代码本大小无质量损失。
- 适用于多种模型结构，特别适合简化下游语音合成建模空间。
基于向量量化变分自编码器（VQ-VAEs）的神经语音编码器是语音大模型的核心音频分词器，但其重建保真度受限于量化误差。传统方法如修改量化器或增加模型容量会复杂化下游语言建模。本文提出自引导机制，通过轻量级特征映射损失，对齐解码器在处理量化令牌及其原始连续嵌入时的内部特征流形。该方法训练开销极小，无需推理阶段改动。应用于XCodec2后，显著提升所有重建指标，在低比特率下达到当前最优表现。尤其可实现4倍代码本缩减而保持保真度，下游文语转换实验表明此举大幅改善基于大模型的语音合成效果，因简化了令牌建模空间。多项统计分析与可视化验证了解码器内部流形对齐的增强。大量实验证实其在不同归纳偏置下的通用性。自引导为高保真神经音频编码提供了一种高效、普适的新方法。
原文摘要 · Abstract (English)
Neural speech codecs based on Vector-Quantized VAEs (VQ-VAEs) are core audio tokenizers for speech LLMs, yet their reconstruction fidelity is bottlenecked by quantization error. Modifying the quantizer or increasing model capacity are common fixes, but they complicate downstream language modeling. Our core idea is to align the decoder's internal feature manifolds when processing both the quantized tokens and their original continuous embeddings, using a lightweight feature-mapping loss. This requires minimal training overhead and no inference-time changes. Applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance. Notably, it enables a 4x codebook reduction without fidelity loss, which downstream TTS experiments show significantly improves LLM-based synthesis by simplifying the token modeling space. Multiple statistical observations and visualizations corroborate the enhanced internal manifold alignment in the decoder. Extensive experiments confirm its generality across various inductive biases. Self-guidance thus establishes an efficient, broadly applicable method for high-fidelity neural audio coding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。