在压缩表示空间直接处理音频，速度提升100倍且质量不降。
Learning to Upsample and Upmix Audio in the Latent Domain
- 在自编码器的潜空间内完成音频升采样与升混音
- 计算效率提升最高达100倍，质量媲美原始波形后处理
- 适合已有自编码器架构的音频任务加速
神经音频自编码器生成紧凑的潜空间表示，保留感知关键信息，是现代音频压缩和生成（如下一步词预测、潜扩散）的基础。然而，大多数音频处理操作（如空间与频谱升采样）仍低效地作用于原始波形或频谱表示，而非直接在压缩表示上进行。本文提出一个完全在自编码器潜空间内执行音频处理的框架，无需解码至原始音频格式。该方法通过仅使用潜空间L1重建项与单一潜空间对抗判别器，大幅简化训练过程，相较原始音频方法所需的多尺度损失与复杂判别器组合有显著优势。在带宽扩展与单声道转立体声升混音任务中，实验表明计算效率最高提升100倍，同时保持与原始音频后处理相当的质量。本工作为已采用自编码器的音频处理流程建立更高效的新范式，实现跨多种音频任务的显著加速与资源节约。
原文摘要 · Abstract (English)
Neural audio autoencoders create compact latent representations that preserve perceptually important information, serving as the foundation for both modern audio compression systems and generation approaches like next-token prediction and latent diffusion. Despite their prevalence, most audio processing operations, such as spatial and spectral up-sampling, still inefficiently operate on raw waveforms or spectral representations rather than directly on these compressed representations. We propose a framework that performs audio processing operations entirely within an autoencoder's latent space, eliminating the need to decode to raw audio formats. Our approach dramatically simplifies training by operating solely in the latent domain, with a latent L1 reconstruction term, augmented by a single latent adversarial discriminator. This contrasts sharply with raw-audio methods that typically require complex combinations of multi-scale losses and discriminators. Through experiments in bandwidth extension and mono-to-stereo up-mixing, we demonstrate computational efficiency gains of up to 100x while maintaining quality comparable to post-processing on raw audio. This work establishes a more efficient paradigm for audio processing pipelines that already incorporate autoencoders, enabling significantly faster and more resource-efficient workflows across various audio tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。