让自回归音频生成更听指令,靠早期预测优化生成路径。
Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decoding
- 用轻量模型提前预测生成质量,指导后续选择
- 比基线提升10分CLAP得分,保持计算开销不变
- 适合追求精准语音指令响应的生成任务
自回归(AR)模型通过逐个生成音素实现时间上连贯的音频生成,但在遵循复杂文本提示时表现不佳,尤其在描述复杂声学事件时。我们发现AR音频生成器的一个意外能力:其早期前缀标记隐含地编码了最终输出的全局语义属性,如事件数量和声音类别,表现出一种隐式规划。基于此,我们提出Plan-Critic——一个使用广义优势估计(GAE)启发目标训练的轻量辅助模型,可从部分生成中预测最终指令遵循质量。推理时,Plan-Critic实现引导式探索:早期评估候选前缀,剔除低保真轨迹,并将计算资源重新分配给高潜力规划种子。该方法在标准best-of-N解码计算成本下,使CLAP得分相比基线最高提升10点,确立了自回归文本到音频生成的新基准。本工作弥合了因果生成与全局语义对齐之间的差距,证明即使严格自回归模型也能进行前瞻规划。
原文摘要 · Abstract (English)
Autoregressive (AR) models excel at generating temporally coherent audio by producing tokens sequentially, yet they often falter in faithfully following complex textual prompts, especially those describing complex sound events. We uncover a surprising capability in AR audio generators: their early prefix tokens implicitly encode global semantic attributes of the final output, such as event count and sound-object category, revealing a form of implicit planning. Building on this insight, we propose Plan-Critic, a lightweight auxiliary model trained with a Generalized Advantage Estimation (GAE)-inspired objective to predict final instruction-following quality from partial generations. At inference time, Plan-Critic enables guided exploration: it evaluates candidate prefixes early, prunes low-fidelity trajectories, and reallocates computation to high-potential planning seeds. Our Plan-Critic-guided sampling achieves up to a 10-point improvement in CLAP score over the AR baseline-establishing a new state of the art in AR text-to-audio generation-while maintaining computational parity with standard best-of-N decoding. This work bridges the gap between causal generation and global semantic alignment, demonstrating that even strictly autoregressive models can plan ahead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。