arXiv:2605.06407eess.AScs.AI2026-05被引 3

用统一编码同时实现语音理解与生成,压缩后仍保持高性能。

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

论文配图:WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
图 1 · 摘自论文原文
  • 分两阶段训练:先过滤冗余语义信息,再注入精细声学细节。
  • 维度压缩8倍下,语音理解性能接近WavLM,零样本语音合成领先。
  • 适合研究统一语音模型、语音生成与重建的学者和开发者。

将语音理解与生成融合是构建统一语音模型的关键一步,但两者所需的表征不同,导致兼容性挑战。通常,语义特征由自监督学习(SSL)获得,声学特征通过重建学习,这种碎片化表征阻碍了真正统一系统的实现。本文提出WavCube,一种从SSL语音编码器导出的紧凑连续潜在表示,可同时支持语音理解、重建与生成。WavCube采用两阶段训练:第一阶段训练语义瓶颈,去除非流形冗余,使原始SSL特征适用于扩散模型;第二阶段通过端到端重建注入细粒度声学细节,同时使用语义锚定损失确保表示保持在原始语义流形内。大量实验表明,尽管维度压缩8倍,WavCube在SUPERB上性能接近WavLM;重建质量与现有声学表示相当;零样本语音合成表现达到当前最优,且训练收敛显著更快;在语音增强、分离与变声任务上,于SUPERB-SG基准上表现优异。系统性消融实验揭示,WavCube的两阶段方案解决了SSL特征在生成建模中的两个内在缺陷,为未来统一语音系统铺平道路。代码与检查点见https://github.com/yanghaha0908/WavCube。

原文摘要 · Abstract (English)

Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.

语音生成统一模型自监督学习扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。