实现音乐伴奏的实时生成，让模型边听边伴奏。
Streaming Generation for Music Accompaniment
- 设计延迟感知的流式生成框架，调节未来可见性和输出块长度。
- 增大未来可见性提升伴奏连贯性，但需更快推理满足延迟要求。
- 传统最大似然训练不足，需引入前瞻与主动生成目标。
音乐生成模型能在完整音频输入下生成高保真、连贯的伴奏，但仅限于编辑和循环工作流。本文研究实时音视频伴奏生成：当模型听到输入音频流（如歌手演唱）时，需同步生成连贯的伴奏流（如吉他伴奏）。为此，我们提出一种考虑实际部署中不可避免系统延迟的模型设计，包含两个变量：未来可见性 $t_f$（输出播放时间与最新输入时间的偏移）、输出块长度 $k$（每次调用输出的帧数）。我们在 $(t_f,k)$ 网格上训练 Transformer 解码器，发现两个稳定权衡：增加有效 $t_f$ 可通过减少近期差距提升连贯性，但需要更快推理以维持延迟预算；增大 $k$ 虽提升吞吐量，却因更新率降低导致伴奏质量下降。最后，我们发现朴素的最大似然流式训练在缺乏未来上下文时不足以生成连贯伴奏，从而激发对实时合奏所需的前瞻与代理目标的改进。
原文摘要 · Abstract (English)
Music generation models can produce high-fidelity coherent accompaniment given complete audio input, but are limited to editing and loop-based workflows. We study real-time audio-to-audio accompaniment: as a model hears an input audio stream (e.g., a singer singing), it has to also simultaneously generate in real-time a coherent accompanying stream (e.g., a guitar accompaniment). In this work, we propose a model design considering inevitable system delays in practical deployment with two design variables: future visibility $t_f$, the offset between the output playback time and the latest input time used for conditioning, and output chunk duration $k$, the number of frames emitted per call. We train Transformer decoders across a grid of $(t_f,k)$ and show two consistent trade-offs: increasing effective $t_f$ improves coherence by reducing the recency gap, but requires faster inference to stay within the latency budget; increasing $k$ improves throughput but results in degraded accompaniment due to a reduced update rate. Finally, we observe that naive maximum-likelihood streaming training is insufficient for coherent accompaniment where future context is not available, motivating advanced anticipatory and agentic objectives for live jamming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。