一个兼顾理解与生成的音频分词器，让声音能被分析也能被重建。
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
- 用噪声正则化瓶颈和潜在编码器改造连续音频自编码器
- 生成目标为可控制尺度的连续潜在表示，支持高质量重建
- 适合需要统一理解与生成的语音模型研究者
连续音频自编码器虽能良好重构波形，但其潜在表示结构较弱，不利于理解；而自监督音频编码器虽捕捉语义，却无法直接解码。这种差异使同时支持理解和生成的单一音频分词器难以实现。本文提出F3-Tokenizer，通过两个组件解决该问题：一是采用通道归一化与随机扰动替代基于KL的变分训练的噪声正则化瓶颈，生成可控制尺度的连续潜在表示，用于重建与自回归生成；二是使用冻结的自编码器潜在表示，结合RQ-MTP与冻结大语言模型监督训练潜在侧表示编码器。最终的分词器既能提供高维语义表示以支持理解，又保留归一化的连续潜在表示作为生成目标。
原文摘要 · Abstract (English)
Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。