EntangleCodec统一建模音频语义与声学特征,实现高质量重建与理解。
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

- 通过歌词对齐学习语义-声学联合表示,再进行量化。
- 在音频理解上比基线高7.4%,重建质量媲美专用编码器。
- 适合需要生成与理解一体的音频大模型研究者使用。
音频分词器作为连续音频与音频语言模型(ALMs)之间的离散接口,现有方法在理解与生成任务间难以兼顾。重建导向的编码器保有声学保真度但语义贫乏,而语义感知分词器通常依赖分离的语义与声学流,带来冗余或错位。本文提出EntangleCodec,一种在量化前学习与标题对齐的语义-声学联合表征的统一离散音频分词器。通过与丰富标题对齐而非自动语音识别转录,该方法在紧凑的标记序列中捕捉语言内容、说话人身份、情感、韵律及声学场景。结合流匹配扩散解码器,实现语音、音乐与通用音频的高质量重建。EntangleCodec在重建质量上可与专用编码器竞争,在音频理解任务上相比所有基于编码器的基线最高提升7.4%(MMAR),并统一支持文本转语音(TTS)与文本到音频生成(TTA)。基于EntangleCodec的音频语言模型展现出强可扩展性:即使仅0.6B参数,其性能也超越超过13B参数的连续表示语言模型(22×更少参数);扩展至8B参数后,在MMAR上刷新了新纪录,表明表征质量与模型规模同样关键。
原文摘要 · Abstract (English)
Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation. Reconstruction-oriented codecs preserve acoustic fidelity but lack rich semantics, while semantic-aware tokenizers typically rely on separate semantic and acoustic streams, introducing redundancy or misalignment. We propose \textbf{EntangleCodec}, a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. By aligning audio with rich captions rather than ASR transcripts, EntangleCodec captures linguistic content, speaker identity, emotion, prosody, and acoustic scenes within a compact token stream. A flow-matching diffusion decoder further enables high-quality reconstruction across speech, music, and general audio. EntangleCodec achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding by up to \textbf{+7.4\%} on MMAR, and supports both TTS and TTA generation in a unified framework. Furthermore, EntangleCodec-based audio language models demonstrate strong scaling behavior: even at \textit{0.6B} parameters, the model surpasses specialized continuous-representation LLMs with over \textit{13B} parameters across three benchmarks using \textbf{22$\times$} fewer parameters; scaling to \textit{8B} further establishes new state-of-the-art results on MMAR, highlighting that representation quality is as critical as model scale in audio language modeling. Code and model weights are available at https://github.com/luckyerr/EntangleCodec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。