用多轮自然语言指令逐步优化音频效果,让声音更贴合创作意图。
InstructFX2FX: A Multi-Turn Text-to-Effect System for Sequential Audio Effect Refinement

- 分层设计:大模型规划效果选择,CLAP引导优化精细调整参数。
- 9/10场景下优化效果优于纯大模型重提示,音质更稳定。
- 适合音乐制作人、音频工程师做渐进式音效迭代。
我们提出 InstructFX2FX,一个通过多轮自然语言指令实现音频效果序列化精炼的系统。现有文本转效果系统多为单次映射,将一句描述直接对应一个预设效果;而真实音频工程是迭代过程:工程师基于已有效果链,逐次添加指令进行优化。这带来状态依赖问题——在当前效果参数基础上,如何根据新指令调整声音,同时保留先前已达成的效果?InstructFX2FX 采用混合架构:大语言模型(LLM)负责高层规划,选择合适效果并提出初始参数;随后由 CLAP 引导的优化模块对现有参数进行精细化调整,相比重复调用大模型更具稳定性。在 SocialFX 衍生的描述对上进行初步评估,9/10 对比中,CLAP 引导优化的数字信号处理特征均方差(MMD)更低。轨迹分析显示,对可微分效果,优化过程会逐步逼近目标音色,同时保留前序指令效果,展现渐进式精炼潜力。
原文摘要 · Abstract (English)
We present InstructFX2FX, a system for sequential audio effect refinement through multi-turn natural-language instructions. Existing text-to-effect systems are largely single-shot, mapping one textual descriptor to one preset. Real audio engineering is instead sequential: engineers refine an existing effect chain through successive instructions. This poses a stateful problem that single-shot systems do not address: given the current effect parameters state and a new instruction, update the sound while preserving what earlier instructions already achieved. InstructFX2FX addresses this with a hybrid architecture that divides labor between a language model and CLAP-guided optimization. The LLM serves as a high-level planner that selects effects and proposes the initial parameter state, motivated by recent evidence that LLMs can outperform CLAP-based optimization for single-turn text-to-effect mapping; CLAP-guided optimization then refines the existing parameter state, providing a more stable and robust refinement mechanism than LLM reprompting. In the demo, attendees drive a dry recording through successive natural-language instructions: after each turn, they choose how strongly the effect is applied, then issue the next instruction based on what still differs from the sound they intend. In a preliminary evaluation on SocialFX-derived descriptor pairs, CLAP-guided refinement achieves lower DSP-feature MMD than an LLM+LLM initialize-then-reprompt baseline on 9 of 10 pairs. Trajectory analysis further shows that, for differentiable effects, optimization tends to gradually move the audio toward the new target while retaining the effects of the previous instruction, highlighting the potential for gradual refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。