无需训练即可精准控制音频时间，生成长时序文本到音频内容
FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generation
- 用大模型规划时间窗口并重写描述，实现精准时间控制
- 在24秒长音频上达到领先训练无关方法的合成质量
- 适合需要精确时间控制的音频生成场景，如影视音效制作
文本到音频（T2A）生成因生成模型进步已取得显著进展。然而，受限于时序对齐的音视频数据质量和数量，现有方法难以处理包含精确时间控制的复杂文本提示，例如“猫头鹰在2.4秒至5.2秒鸣叫”。近期工作尝试通过数据增强或引入时间条件作为模型输入来实现10秒内的时序控制生成，但合成质量仍有限。本文提出一种全新的无训练时序控制T2A框架FreeAudio，首次实现时序控制的长时序T2A生成，例如“猫头鹰在2.4秒-5.2秒鸣叫，蟋蟀在0秒-24秒持续鸣叫”。具体而言，我们首先利用大语言模型（LLM）根据输入文本与时间提示规划非重叠时间窗口，并为每个窗口生成优化后的自然语言描述。随后引入：1）解耦聚合注意力控制以实现精确时间控制；2）上下文潜在体组合以保证局部平滑性，参考引导以维持全局一致性。大量实验表明：1）FreeAudio在训练无关方法中达到最先进的时序控制音频生成质量，且与领先的训练依赖方法相当；2）FreeAudio在长时序生成方面表现可媲美训练型Stable Audio，为时序控制的长时序T2A合成铺平道路。演示样本见：https://freeaudio.github.io/FreeAudio/
原文摘要 · Abstract (English)
Text-to-audio (T2A) generation has achieved promising results with the recent advances in generative models. However, because of the limited quality and quantity of temporally-aligned audio-text pairs, existing T2A methods struggle to handle the complex text prompts that contain precise timing control, e.g., "owl hooted at 2.4s-5.2s". Recent works have explored data augmentation techniques or introduced timing conditions as model inputs to enable timing-conditioned 10-second T2A generation, while their synthesis quality is still limited. In this work, we propose a novel training-free timing-controlled T2A framework, FreeAudio, making the first attempt to enable timing-controlled long-form T2A generation, e.g., "owl hooted at 2.4s-5.2s and crickets chirping at 0s-24s". Specifically, we first employ an LLM to plan non-overlapping time windows and recaption each with a refined natural language description, based on the input text and timing prompts. Then we introduce: 1) Decoupling and Aggregating Attention Control for precise timing control; 2) Contextual Latent Composition for local smoothness and Reference Guidance for global consistency. Extensive experiments show that: 1) FreeAudio achieves state-of-the-art timing-conditioned T2A synthesis quality among training-free methods and is comparable to leading training-based methods; 2) FreeAudio demonstrates comparable long-form generation quality with training-based Stable Audio and paves the way for timing-controlled long-form T2A synthesis. Demo samples are available at: https://freeaudio.github.io/FreeAudio/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。