用大模型生成连贯长篇叙事音频，让故事有逻辑、有情感。
AudioStory: Generating Long-Form Narrative Audio with Large Language Models
- 用大模型拆解复杂故事指令，分步生成有时间顺序的音频场景。
- 在多个任务上超越现有方法，在指令遵循和音质上表现更优。
- 适合做有声书、动画配音等需要长篇连贯音频的场景。
文本到音频（TTA）生成近期进展在短音频合成上表现良好，但在长篇叙事音频方面仍面临时间连贯性与结构推理挑战。为此，我们提出AudioStory，一个统一框架，将大语言模型（LLMs）与TTA系统结合，生成结构化、长篇音频叙事。AudioStory具备强指令遵循与推理生成能力，利用LLM将复杂叙事请求分解为具有上下文提示的时序子任务，实现连贯的场景转换与情感基调一致。其两大优势：（1）解耦桥接机制：将LLM与扩散模型协作分为两个专用组件——事件内语义对齐的桥接查询与事件间连贯性保持的残差查询；（2）端到端训练：在单一端到端框架中统一指令理解与音频生成，无需模块化训练流程，增强组件协同。此外，我们构建了基准数据集AudioStory-10K，涵盖动画音景与自然声音叙事等多元领域。大量实验表明，AudioStory在单音频生成与叙事音频生成任务上均优于现有基线，在指令遵循能力和音频保真度方面表现更佳。代码已开源于https://github.com/TencentARC/AudioStory。
原文摘要 · Abstract (English)
Recent advances in text-to-audio (TTA) generation excel at synthesizing short audio clips but struggle with long-form narrative audio, which requires temporal coherence and compositional reasoning. To address this gap, we propose AudioStory, a unified framework that integrates large language models (LLMs) with TTA systems to generate structured, long-form audio narratives. AudioStory possesses strong instruction-following reasoning generation capabilities. It employs LLMs to decompose complex narrative queries into temporally ordered sub-tasks with contextual cues, enabling coherent scene transitions and emotional tone consistency. AudioStory has two appealing features: (1) Decoupled bridging mechanism: AudioStory disentangles LLM-diffuser collaboration into two specialized components, i.e., a bridging query for intra-event semantic alignment and a residual query for cross-event coherence preservation. (2) End-to-end training: By unifying instruction comprehension and audio generation within a single end-to-end framework, AudioStory eliminates the need for modular training pipelines while enhancing synergy between components. Furthermore, we establish a benchmark AudioStory-10K, encompassing diverse domains such as animated soundscapes and natural sound narratives. Extensive experiments show the superiority of AudioStory on both single-audio generation and narrative audio generation, surpassing prior TTA baselines in both instruction-following ability and audio fidelity. Our code is available at https://github.com/TencentARC/AudioStory
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。