一个音视频统一的连续分词器，仅用一层就实现听觉理解与生成。
DashengTokenizer: One layer is enough for unified audio understanding and generation
- 用冻结语义特征注入声学信息，反向构建分词器。
- 22项任务线性评估中表现优于现有音频编码器，重建质量不降。
- 适合语音情感识别、音乐理解等多场景应用，可替代变分自编码器。
本文提出DashengTokenizer，一种专为联合理解与生成任务设计的连续音频分词器。与传统方法先训练声学分词器再冻结语义知识不同，本方法反其道而行之：利用冻结的语义特征并注入声学信息。在22个多样化任务的线性评估中，该方法显著超越现有音频编解码器和音频编码器基线，同时保持良好的音频重建质量。特别地，声学注入有效提升了语音情感识别、音乐理解及声学场景分类等任务的性能。我们在文本到音频（TTA）、文本到音乐（TTM）和语音增强（SE）任务上进一步评估了生成能力，结果表明其在TTA与TTM任务上优于标准变分自编码器（VAE）方法，且在SE任务中的有效性凸显其作为通用音频编码器的潜力。研究挑战了当前认为VAE架构是音频合成前提的普遍假设。模型权重已公开于https://huggingface.co/mispeech/dashengtokenizer。
原文摘要 · Abstract (English)
This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。