让语音分词器同时懂语音和环境声,不丢说话质量。
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception

- 用三类结构化监督分解音频:语言内容、发声特征、声音场景。
- 引入自适应门控机制,从浅层恢复精细声学细节。
- 兼容现有语音模型,提升听觉理解与生成效果。
语义语音分词器因紧凑的单码本设计和强语言对齐性,已成为音频大模型的通用接口。但其过度关注语言抽象,导致对非语音信息的感知缺失,限制了在非语音任务中的应用。我们提出UniAudio-Token框架,在不改变原有语义范式的基础上,通过两项创新赋予语义分词器通用音频感知能力:(1) 语义-声学基元(SAP)通过将音频分解为语言内容、发声属性和听觉场景基元,提供结构化监督;(2) 语义-声学平衡(SAE)引入内容感知门控机制,自适应地从浅层恢复细粒度声学细节。大量实验表明,UniAudio-Token在保持高保真语音生成的同时,学习到全面的通用音频表征。当与下游大语言模型集成时,其在理解和生成任务上均优于所有单码本基线分词器,可作为统一的音频接口。代码与模型权重已开源于https://github.com/Tencent/Universal_Audio_Tokenizer。
原文摘要 · Abstract (English)
Semantic speech tokenizers have become a widely used interface for Audio-LLMs, owing to their compact single-codebook design and strong linguistic alignment. However, their focus on linguistic abstraction induces acoustic blindness, limiting their applicability beyond speech-centric tasks. We propose UniAudio-Token, a framework that empowers semantic tokenizers with general audio perception without compromising speech ability. Instead of altering the semantic paradigm, UniAudio-Token mitigates its information loss through two key innovations: (1) Semantic-Acoustic Primitives (SAP) provide structured supervision by decomposing audio into linguistic content, vocal attributes, and auditory-scene primitives; and (2) Semantic-Acoustic Equilibrium (SAE) introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers. Extensive evaluations show that UniAudio-Token learns comprehensive universal representations while preserving high-fidelity speech generation. When integrated with downstream LLMs, it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks, effectively serving as a unified audio interface. We publicly release all our code, including training and inference scripts, together with the model checkpoints at https://github.com/Tencent/Universal_Audio_Tokenizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。