arXiv:2605.27840eess.AScs.AI2026-05被引 2

用128维低维表示统一音频理解与生成,提升扩散模型效率。

LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation

论文配图:LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation
图 1 · 摘自论文原文
  • 将1280维语义特征压缩至128维,引入时间关系损失保持时序一致性。
  • 在语音、音乐和通用音频上,理解性能接近先进语义表示,生成效果更优。
  • 适合需要高效跨域音频建模的场景,如语音合成与音乐生成。

音频分词器是统一音频理解与生成的基础。理解需高层语义,生成需语义与声学细节。现有统一分词器在高维连续潜在空间中联合编码,增加了扩散变压器(DiTs)的建模负担。我们提出LoSATok,一种用于跨域音频理解与生成的低维音频分词器。基于1280维语义编码器特征可压缩的观察,我们引入语义瓶颈(SemBo),将其压缩至128维,并通过提出的时序关系损失正则化以保持时序特征一致性。进一步设计双层语义监督机制,同时利用高维与低维语义信号,使分词器在紧凑潜在空间中联合捕捉语义与声学细节。在语音、音乐和通用音频上的实验表明,SemBo保留强低维语义能力,LoSATok在理解性能上与多个语义表示相当,且在语音、音乐和音频生成任务中持续提升DiT建模性能。结果证明,LoSATok的低维表示能有效支持音频理解与生成。代码已开源:https://github.com/wxzyd123/LoSATok。

原文摘要 · Abstract (English)

Audio tokenizers are fundamental to unifying audio understanding and generation. Understanding requires high-level semantics, while generation demands semantic and acoustic details. Existing unified tokenizers jointly encode both in high-dimensional continuous latents, which increases the modeling burden of Diffusion Transformers (DiTs) for generation. We propose LoSATok, a low-dimensional audio tokenizer for cross-domain audio understanding and generation. Motivated by the observation that 1280-dimensional semantic encoder features are compressible, we introduce a Semantic Bottleneck that compresses them into 128 dimensions, regularized by the proposed time-relation loss for temporal feature consistency. We further design a dual-level semantic supervision method that leverages both high- and low-dimensional semantic signals, enabling the tokenizer to jointly capture semantics and acoustic details within a compact latent space. Experiments on speech, music, and general audio show that SemBo preserves strong low-dimensional semantic capacity and LoSATok retains competitive understanding performance compared with several semantic representations, while consistently improving DiT modeling performance on speech, music, and audio generation. These results demonstrate that LoSATok's low-dimensional representations can effectively support audio understanding and generation. Our code is provided at https://github.com/wxzyd123/LoSATok.

音频生成低维表示扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。