用扩散模型实现高保真小提琴合成，支持演奏技巧与动态控制。
VIOLET: High-Fidelity Violin Synthesis with Techniques and Dynamics
- 基于扩散Transformer的潜空间建模，融合MIDI、技巧与连续动态输入
- 在39小时合成音频上训练，实现高精度音高/节奏对齐与技巧还原
- 效果接近顶级商业虚拟乐器，适合音乐创作与真实演奏模拟
神经音频合成有望革新当前依赖拼接合成与采样库的音乐制作方式。然而，现有研究多集中于钢琴合成与表现力生成，对小提琴等持续发声乐器，尤其是其演奏技巧与动态变化的建模仍鲜有探索。本文提出VIOLET，一种基于潜空间扩散框架的小提琴可控合成方法，采用带有修正流的扩散Transformer（DiT），从MIDI音符、演奏技巧和连续动态曲线生成高保真音频。为训练VIOLET，除使用现有数据集外，还构建了名为CSV-TD的新数据集，包含39小时48 kHz合成音频及时间对齐的MIDI音符、音级技巧标注与连续动态曲线。客观与主观评估表明，VIOLET能实现高技巧符合度、精准音高与节拍对齐，以及良好动态控制，性能超越当前最先进的神经小提琴合成系统，并在技巧清晰度、自然度与动态跟随方面接近顶尖商业虚拟乐器。
原文摘要 · Abstract (English)
Neural synthesis for musical instruments has the potential to revolutionize current practices that use concatenative synthesis and a sample library. However, most research focused on piano synthesis and expressive performance generation; little work has been done on continuously articulated instruments like the violin, let alone rendering them with playing techniques and dynamics. We present VIOLET, a latent-diffusion framework for controllable violin synthesis, which uses a Diffusion Transformer (DiT) with rectified flow to synthesize high-fidelity audio from MIDI notes, playing techniques, and continuous dynamics. To train VIOLET, in addition to using a few existing datasets, we curate a new dataset named CSV-TD, which contains 39 h of 48 kHz synthetic audio and time-aligned annotations of MIDI notes, note-level techniques, and continuous dynamics curves. Objective and subjective evaluations show that VIOLET synthesizes violin performances with high technique adherence, accurate pitch and timing alignment, and good dynamics control. It outperforms the current state-of-the-art neural violin synthesis system and approaches a top commercial virtual instrument in terms of technique clarity, naturalness, and dynamics following.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。