低资源实现音频扩散模型的精细控制，计算开销更低。
Low-Resource Guidance for Controllable Latent Audio Diffusion
- 在潜在空间直接控制，避免解码器反向传播耗时
- 仅需700万参数和4小时训练，支持音高、节奏等多维控制
- 适合资源有限但需精准调控的音频生成场景
生成式音频需要细粒度可控输出，但现有方法要么需针对特定控制重新训练模型，要么在推理时使用引导机制，计算成本较高。通过分析现有基于引导控制的瓶颈，特别是每步计算开销大（源于解码器反向传播），我们提出一种基于选择性TFG和潜在控制头（LatCHs）的引导方法，可在低计算开销下实现潜在音频扩散模型的可控生成。LatCHs 直接在潜在空间操作，跳过昂贵的解码步骤，且仅需700万参数和约4小时训练。在 Stable Audio Open 上的实验表明，该方法能有效控制音量、音高和节拍（及组合），同时保持生成质量。相比端到端引导，本方法在精度与音频保真度间取得更好平衡，计算成本大幅降低。演示示例见：https://zacharynovack.github.io/latch/latch.html。
原文摘要 · Abstract (English)
Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (\textit{e.g.}, guidance) that can also be computationally demanding. By examining the bottlenecks of existing guidance-based controls, in particular their high cost-per-step due to decoder backpropagation, we introduce a guidance-based approach through selective TFG and Latent-Control Heads (LatCHs), which enables controlling latent audio diffusion models with low computational overhead. LatCHs operate directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and $\approx$ 4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. Our method balances precision and audio fidelity with far lower computational costs than standard end-to-end guidance. Demo examples can be found at https://zacharynovack.github.io/latch/latch.html.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。