提出低延迟神经音频合成模型，让音乐人实时互动更流畅。
Designing Neural Synthesizers for Low-Latency Interaction
- 分析延迟来源并迭代优化架构，提升实时性。
- 新模型BRAVE实现低延迟，音高响度还原更准。
- 专为音乐交互设计，适合开发者和作曲家使用。
神经音频合成（NAS）模型可实现高质量、富有表现力的音频生成与实时音乐控制。尽管这些模型具备实时运行能力，但通常存在高延迟问题，难以支持亲密的音乐互动。本文研究了交互式NAS模型中常见的延迟与抖动来源，并将其应用于基于RAVE（Caillon等，2021年提出的卷积变分自编码器）的音色迁移任务。通过迭代设计优化，我们提出名为BRAVE（Bravely Realtime Audio Variational autoEncoder）的新模型，其延迟更低，且在音高与响度还原方面表现更优，同时保持与RAVE相当的音色修改能力。我们构建了专用推理框架以支持低延迟实时推理，并实现了一个兼容乐器信号的音频插件原型。本文提出的挑战与设计指南可帮助NAS研究者从零开始构建低延迟模型，拓展音乐创作的可能性。
原文摘要 · Abstract (English)
Neural Audio Synthesis (NAS) models offer interactive musical control over high-quality, expressive audio generators. While these models can operate in real-time, they often suffer from high latency, making them unsuitable for intimate musical interaction. The impact of architectural choices in deep learning models on audio latency remains largely unexplored in the NAS literature. In this work, we investigate the sources of latency and jitter typically found in interactive NAS models. We then apply this analysis to the task of timbre transfer using RAVE, a convolutional variational autoencoder for audio waveforms introduced by Caillon et al. in 2021. Finally, we present an iterative design approach for optimizing latency. This culminates with a model we call BRAVE (Bravely Realtime Audio Variational autoEncoder), which is low-latency and exhibits better pitch and loudness replication while showing timbre modification capabilities similar to RAVE. We implement it in a specialized inference framework for low-latency, real-time inference and present a proof-of-concept audio plugin compatible with audio signals from musical instruments. We expect the challenges and guidelines described in this document to support NAS researchers in designing models for low-latency inference from the ground up, enriching the landscape of possibilities for musicians.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。