用音符和文本生成音乐，无需微调即可克隆乐器音色
TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument
- 基于MIDI和文本嵌入，用纯解码器变压器生成音频标记
- 合成音质高，音色相似度达90%以上，准确跟随输入音符
- 适合音乐制作人、音色设计师快速实现文本控音
神经音频编码器的进步使得音频标记表示可用于多种音频生成任务，如文到语音、文到音频及文到音乐生成。我们提出TokenSynth，一种新型神经合成器，利用仅解码器的Transformer模型，从MIDI标记和包含音色信息的CLAP嵌入中生成目标音频标记。该模型无需微调即可实现乐器克隆、文到乐器合成及文本引导的音色操控，具备高度灵活性，支持多样化的音色设计与直观音色控制。我们通过客观指标评估了合成音频质量、合成音色与目标音色/文本的相似度以及合成准确性（即是否准确遵循输入MIDI）。实验表明，该模型充分展现了先进神经音频编码器与Transformer结合在构建强大且多功能神经合成器方面的潜力。源代码、模型权重及音频演示可于 https://github.com/KyungsuKim42/tokensynth 获取。
原文摘要 · Abstract (English)
Recent advancements in neural audio codecs have enabled the use of tokenized audio representations in various audio generation tasks, such as text-to-speech, text-to-audio, and text-to-music generation. Leveraging this approach, we propose TokenSynth, a novel neural synthesizer that utilizes a decoder-only transformer to generate desired audio tokens from MIDI tokens and CLAP (Contrastive Language-Audio Pretraining) embedding, which has timbre-related information. Our model is capable of performing instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation without any fine-tuning. This flexibility enables diverse sound design and intuitive timbre control. We evaluated the quality of the synthesized audio, the timbral similarity between synthesized and target audio/text, and synthesis accuracy (i.e., how accurately it follows the input MIDI) using objective measures. TokenSynth demonstrates the potential of leveraging advanced neural audio codecs and transformers to create powerful and versatile neural synthesizers. The source code, model weights, and audio demos are available at: https://github.com/KyungsuKim42/tokensynth
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。