无需预训练嵌入，直接用参考音频生成逼真乐器音色。
Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance
- 用未压缩音频和目标乐谱直接引导扩散模型生成音色。
- 在音质、音色相似度和旋律还原度上超越现有方法。
- 支持参考音频越长音色越准，适合需要高保真克隆的场景。
零样本乐器克隆旨在仅凭一段[参考音频, 参考乐谱]对，将任意[目标乐谱]序列以未见过的乐器声学特征演奏出来。现有方法依赖预训练嵌入（如CLAP）将参考音频压缩为固定长度向量，丢失了音色重建至关重要的精细声学线索。我们提出Anysynth，一种基于上下文流匹配的无嵌入神经合成器。通过将扩散Transformer (DiT) 直接条件于未压缩的参考音频与目标乐谱，模型可在生成时动态检索声学细节。实验表明，AnySynth在音频质量、音色相似度和旋律遵循度上优于基于嵌入和自回归的基线方法。值得注意的是，该模型表现出提示长度扩展性：更长的参考提示带来持续提升的音色保真度，这是基于嵌入系统所不具备的特性。为优化可控性，我们进一步提出非对称分层CFG，基于其自然语义-声学依赖关系，结构化地解耦乐谱与参考音色引导。该非对称形式避免梯度冲突，同时提升音符准确率与音色保真度，推动零样本乐器克隆的表现边界。演示音频见https://anysynth-demo.github.io/
原文摘要 · Abstract (English)
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that AnySynth outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。