提出STACodec，让音频编码同时保真音质与语义信息。
STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs
- 通过语义令牌分配，将自监督模型语义融入第一层向量量化。
- 在语音重建和语义任务上均优于现有混合编码器。
- 无需依赖语义分词器，推理更高效，适合语音生成与理解场景。
神经音频编码器广泛用于音频压缩，并可集成到基于标记的语言模型中。传统编码器保留良好声学细节，但缺乏语义信息。近期混合编码器尝试通过知识蒸馏引入语义信息，但常导致重建性能下降，难以兼顾两者。为此，我们提出STACodec，一种统一编码器，通过语义令牌分配（STA）将自监督学习（SSL）模型的语义信息整合至残差向量量化（RVQ-1）的第一层。为进一步减少对基于SSL的语义分词器的依赖并提升推理效率，我们设计了语义预蒸馏（SPD）模块，可在推理时直接预测语义令牌并分配给第一层RVQ。实验表明，STACodec在音频重建与下游语义任务上均优于现有混合编码器，实现了声学保真度与语义能力的更好平衡。
原文摘要 · Abstract (English)
Neural audio codecs are widely used for audio compression and can be integrated into token-based language models. Traditional codecs preserve acoustic details well but lack semantic information. Recent hybrid codecs attempt to incorporate semantic information through distillation, but this often degrades reconstruction performance, making it difficult to achieve both. To address this limitation, we introduce STACodec, a unified codec that integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1) via semantic token assignment (STA). To further eliminate reliance on SSL-based semantic tokenizers and improve efficiency during inference, we propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference. Experimental results show that STACodec outperforms existing hybrid codecs in both audio reconstruction and downstream semantic tasks, demonstrating a better balance between acoustic fidelity and semantic capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。