用扩散模型实现实时人机音乐合奏,低延迟响应演奏者即兴伴奏。
Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSP
- 采用滑动窗口前瞻机制,根据部分上下文预测未来音频。
- 通过一致性蒸馏将采样时间减少5.4倍,实现真正实时运行。
- 适合音乐创作者与交互式演出,突破传统工具与AI模型的隔阂。
我们提出一个面向实时人机音乐协作的框架,利用潜在扩散模型根据实时输入音频流生成乐器伴奏。系统结合MAX/MSP前端处理实时音频输入、缓冲与回放,并通过OSC/UDP消息与运行生成模型的Python推理服务器通信。该设计使音乐家可在成熟且支持实时处理的MAX/MSP环境中演奏,同时与基于Python的大规模生成模型互动,克服了实时音乐工具与前沿AI模型之间的根本性断层。我们将伴奏生成建模为滑动窗口前瞻协议,训练模型从部分上下文预测未来音频,其中系统延迟是关键约束。为降低延迟,对扩散模型应用一致性蒸馏,实现5.4倍采样时间减少,两个模型均达到实时运行。在音乐连贯性、节拍对齐和音频质量方面评估,两者在回溯模式下表现良好,且随着前瞻深度增加仍能平稳退化。结果表明基于扩散模型的实时伴奏可行,并揭示了延迟、前瞻深度与生成质量间的根本权衡,任何此类系统都需面对这一挑战。
原文摘要 · Abstract (English)
We present a framework for real-time human-AI musical co-performance, in which a latent diffusion model generates instrumental accompaniment in response to a live stream of context audio. The system combines a MAX/MSP front-end-handling real-time audio input, buffering, and playback-with a Python inference server running the generative model, communicating via OSC/UDP messages. This allows musicians to perform in MAX/MSP - a well-established, real-time capable environment - while interacting with a large-scale Python-based generative model, overcoming the fundamental disconnect between real-time music tools and state-of-the-art AI models. We formulate accompaniment generation as a sliding-window look-ahead protocol, training the model to predict future audio from partial context, where system latency is a critical constraint. To reduce latency, we apply consistency distillation to our diffusion model, achieving a 5.4x reduction in sampling time, with both models achieving real-time operation. Evaluated on musical coherence, beat alignment, and audio quality, both models achieve strong performance in the Retrospective regime and degrade gracefully as look-ahead increases. These results demonstrate the feasibility of diffusion-based real-time accompaniment and expose the fundamental trade-off between model latency, look-ahead depth, and generation quality that any such system must navigate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。