轻量级音乐生成模型，能实时根据人声伴奏作曲。
SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusion
- 用软对齐注意力动态融合局部与全局时间依赖，适应不同生成阶段。
- 仅1500万参数，推理速度比顶尖系统快52倍，参数量减少220倍。
- 适合移动端、交互式应用，可实现在普通电脑上实时生成伴奏。
我们提出一种轻量级潜空间扩散模型，用于人声条件下的音乐伴奏生成，解决了现有音乐AI系统的若干关键局限。该方法引入一种新颖的软对齐注意力机制，根据扩散步骤自适应地结合局部与全局时间依赖，有效捕捉多尺度音乐结构。模型在预训练变分自编码器的压缩潜空间中运行，相比当前最优系统，参数量减少220倍，推理速度提升52倍。实验表明，仅使用1500万参数即可达到竞争力表现，其生成质量与内容一致性优于OpenAI Jukebox，同时保持合理的音乐连贯性。超轻量架构支持在消费级硬件上实时部署，使AI辅助音乐创作适用于交互应用与资源受限环境。
原文摘要 · Abstract (English)
We present a lightweight latent diffusion model for vocal-conditioned musical accompaniment generation that addresses critical limitations in existing music AI systems. Our approach introduces a novel soft alignment attention mechanism that adaptively combines local and global temporal dependencies based on diffusion timesteps, enabling efficient capture of multi-scale musical structure. Operating in the compressed latent space of a pre-trained variational autoencoder, the model achieves a 220 times parameter reduction compared to state-of-the-art systems while delivering 52 times faster inference. Experimental evaluation demonstrates competitive performance with only 15M parameters, outperforming OpenAI Jukebox in production quality and content unity while maintaining reasonable musical coherence. The ultra-lightweight architecture enables real-time deployment on consumer hardware, making AI-assisted music creation accessible for interactive applications and resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。