用提示词控制注意力，实现高效自动语音编辑
EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing
- 通过提示词引导注意力机制实现精准音频编辑
- 在旋律、动态和节奏上优于扩散模型基线
- 适合音乐生成与交互式音频编辑研究者
本文研究在自回归模型中利用交叉注意力控制实现高效的音频编辑。受图像编辑方法启发，我们提出类似Prompt-to-Prompt的方法，通过交叉注意力和自注意力机制引导编辑。结合受Auffusion启发的扩散策略，扩展模型功能以支持细化编辑，并建立提示引导音频编辑的基准。此外，引入预训练冻结的自回归模型MUSICGEN，提出三种基于注意力分数的编辑机制：替换、重加权和细化。采用常用音乐评估指标与人工评测，衡量时变可控性、全局文本提示遵循度及整体音频真实感。自动与人工评估表明，提示词引导结合自回归生成显著优于扩散基线，在旋律、动态和节奏方面表现更优。代码已开源。
原文摘要 · Abstract (English)
In this study, we investigate leveraging cross-attention control for efficient audio editing within auto-regressive models. Inspired by image editing methodologies, we develop a Prompt-to-Prompt-like approach that guides edits through cross and self-attention mechanisms. Integrating a diffusion-based strategy, influenced by Auffusion, we extend the model's functionality to support refinement edits, establishing a baseline for prompt-guided audio editing. Additionally, we introduce an alternative approach by incorporating MUSICGEN, a pre-trained frozen auto-regressive model, and propose three editing mechanisms, based on Replacement, Reweighting, and Refinement of the attention scores. We employ commonly-used music-specific evaluation metrics and a human study, to gauge time-varying controllability, adherence to global text cues, and overall audio realism. The automatic and human evaluations indicate that the proposed combination of prompt-to-prompt guidance with autoregressive generation models significantly outperforms the diffusion-based baseline in terms of melody, dynamics, and tempo of the generated audio. Our code is available at https://github.com/billsioros/EditGen
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。