用离散潜空间构建可解释的语音生成流水线,轻量高效且便于优化。
VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space
- 先用VQ-VAE压缩语音频谱为离散潜码,再用Transformer建模潜码分布
- 在AudioMNIST上生成可懂语音,仅需少量计算资源即可实现高保真度
- 模块化设计使分析与改进更直观,适合研究语音合成机制的开发者
高效生成高质量语音仍是语音合成中生成模型的关键挑战。本文提出VQalAttent,一种轻量级语音生成框架,具备可调节性能与透明性。基于包含0-9数字语音的人类发音数据集AudioMNIST,该方法采用两阶段架构:首先使用可扩展向量量化自编码器(VQ-VAE)将语音频谱压缩为离散潜变量表示;其次,使用仅解码器的Transformer学习这些潜变量的概率模型。训练后的Transformer生成相似潜变量序列,经由VQ-VAE解码器还原为频谱图,进而合成假语音。通过分析潜空间维度及外部信息对生成语音统计与感知质量的影响,可指导更大规模商业生成模型的优化。实验表明,VQalAttent能在有限计算资源下生成可理解的语音样本,其模块化与透明化的训练流程有助于将分析结果与模块化修改直接关联,为复杂模型提供可解释性洞察。
原文摘要 · Abstract (English)
Generating high-quality speech efficiently remains a key challenge for generative models in speech synthesis. This paper introduces VQalAttent, a lightweight model designed to generate fake speech with tunable performance and interpretability. Leveraging the AudioMNIST dataset, consisting of human utterances of decimal digits (0-9), our method employs a two-step architecture: first, a scalable vector quantized autoencoder (VQ-VAE) that compresses audio spectrograms into discrete latent representations, and second, a decoder-only transformer that learns the probability model of these latents. Trained transformer generates similar latent sequences, convertible to audio spectrograms by the VQ-VAE decoder, from which we generate fake utterances. Interpreting statistical and perceptual quality of the fakes, depending on the dimension and the extrinsic information of the latent space, enables guided improvements in larger, commercial generative models. As a valuable tool for understanding and refining audio synthesis, our results demonstrate VQalAttent's capacity to generate intelligible speech samples with limited computational resources, while the modularity and transparency of the training pipeline helps easily correlate the analytics with modular modifications, hence providing insights for the more complex models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。