提出新方法让音频生成模型的隐空间更有序,提升音质和语义保留。
STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation

- 设计生长约束场,按能力分配结构与纹理信息到不同通道子空间
- 在多个音频任务上实现最高重建保真度和更强语义保留
- 适用于各类VAE架构,特别适合文本转音频等生成场景
连续变分自编码器(VAEs)是现代神经音频生成系统的基础连续分词器,可在保持高保真重建的同时提供紧凑平滑的潜在空间以支持后续生成先验。然而,连续VAE面临压缩率、重建保真度与潜在空间拓扑之间的根本矛盾,我们将其形式化为速率-失真-规则性三难困境。该困境源于拓扑不匹配:标准VAE中的各向同性高斯先验施加了平坦的潜在几何结构,无法适应音频的层次特性——低频成分具有结构且可压缩,而高频成分则呈随机性且不可压缩,导致关键语义特征与高熵噪声交织混杂。为此,我们提出结构拓扑感知正则化(STAR),一种通用训练策略,通过施加基于增长的约束场重塑潜在空间几何,将结构性与纹理信息路由至容量匹配的通道子空间。STAR可适配任意VAE架构,并在基于CNN的VAE中有效解决该三难困境。我们进一步提出STAR-VAE,结合混合CNN-Mamba架构实现局部特征提取与线性复杂度全局上下文建模;以及基于大语言模型的流匹配框架STAR-Gen,利用STAR-VAE的结构化潜在空间进行高保真生成,避免向量量化伪影。跨多种音频领域的实验表明,STAR-VAE在重建保真度和语义信息保留方面达到当前最优水平,且结构化潜在空间显著提升传统扩散模型与STAR-Gen在文本到音频生成中的性能。
原文摘要 · Abstract (English)
Continuous Variational Autoencoders (VAEs) serve as the fundamental continuous tokenizer for modern neural audio generation systems, enabling high-fidelity reconstruction while providing a compact, smooth latent space for downstream generative priors. However, continuous VAEs face a fundamental conflict among compression rate, reconstruction fidelity, and latent space topology, which we formalize as the Rate-Distortion-Regularity Trilemma. This trilemma stems from a topological mismatch: the isotropic Gaussian prior in standard VAEs imposes a flat latent geometry that fails to accommodate audio's hierarchical nature, where low-frequency components are structured and compressible while high-frequency components are stochastic and incompressible, leading to disordered information packing in which crucial semantic features are interleaved with high-entropy noise. To address this challenge, we propose Structured Topology-Aware Regularization (STAR), a general training strategy that reshapes latent space geometry by imposing a growth-based constraint field, routing structural and textural information into channel subspaces with matching capacities. STAR is applicable to any VAE architecture and effectively resolves the trilemma, as demonstrated in CNN-based VAEs. We further present STAR-VAE, which combines STAR with a hybrid CNN-Mamba architecture for local feature extraction and linear-complexity global context modeling, and STAR-Gen, an LLM-based Flow Matching framework that leverages STAR-VAE's structured latent space for high-fidelity generation without vector quantization artifacts. Experiments across diverse audio domains show that STAR-VAE achieves state-of-the-art reconstruction fidelity and enhanced semantic information preservation, while the structured latent space improves both traditional diffusion models and STAR-Gen for text-to-audio generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。