用语音语言模型指导立体声音频编辑，实现自然语言指令的智能拆解与执行。
Guiding Audio Editing with Audio Language Model
- 将自然语言指令分解为添加、删除、空间移动等原子操作
- 在立体声场景中实现高质量、空间真实的音频编辑效果
- 适合音视频创作、虚拟现实等需要语义级编辑的场景
音频编辑在虚拟现实、远程会议、声音设计等交互媒体中至关重要。然而，现有生成式音频编辑模型依赖模板化指令，仅支持单通道音频，难以处理陈述性编辑——即用户仅说明期望结果，由系统自主决定操作细节。我们提出 SmartDJ 框架，结合语音语言模型的推理能力与潜在扩散模型的生成能力，实现立体声音频的智能编辑。给定高层指令后，SmartDJ 将其分解为一系列原子编辑操作（如添加、移除、空间重定位事件），并由训练过的扩散模型执行。为此，我们设计数据合成流程，生成包含高层指令、原子操作和编辑前后音频的成对数据。实验表明，SmartDJ 在感知质量、空间真实性和语义一致性上均优于现有方法。演示地址：https://zitonglan.github.io/project/smartdj/smartdj.html。
原文摘要 · Abstract (English)
Audio editing plays a central role in VR/AR immersion, virtual conferencing, sound design, and other interactive media. However, recent generative audio editing models depend on template-like instruction formats and are restricted to mono-channel audio. These models fail to deal with declarative audio editing, where the user declares what the desired outcome should be, while leaving the details of editing operations to the system. We introduce SmartDJ, a novel framework for stereo audio editing that combines the reasoning capability of audio language models with the generative power of latent diffusion. Given a high-level instruction, SmartDJ decomposes it into a sequence of atomic edit operations, such as adding, removing, or spatially relocating events. These operations are then executed by a diffusion model trained to manipulate stereo audio. To support this, we design a data synthesis pipeline that produces paired examples of high-level instructions, atomic edit operations, and audios before and after each edit operation. Experiments demonstrate that SmartDJ achieves superior perceptual quality, spatial realism, and semantic alignment compared to prior audio editing methods. Demos are available at https://zitonglan.github.io/project/smartdj/smartdj.html.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。