从音频生成小提琴演奏动作,还原人琴互动细节。
ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition
- 基于扩散模型,仅凭音频生成全身精细演奏动作。
- 引入接触损失函数,确保手部与琴弦、弓与琴弦的交互真实。
- 设计新评估指标,适合音乐性与动作准确性双重验证。
乐器演奏艺术是人类创造力与情感的生动体现。然而,生成乐器演奏动作极具挑战性,需捕捉复杂动作并重建演奏者与乐器间的动态交互。现有研究多聚焦局部身体动作,本文提出面向音频还原的小提琴演奏动作生成框架ELGAR,采用基于扩散模型的全身体细粒度动作生成方法。为强调演奏的交互特性,引入手部交互接触损失(HICL)和弓部交互接触损失(BICL),有效保障交互真实性。此外,为评估生成动作与音乐语义的一致性,设计了针对弦乐器演奏动作的新评估指标,包括指触距离、弓弦距离与运弓评分。通过大量实验与消融研究验证了方法的有效性。同时,构建了新动作生成数据集SPD-GEN,源自动作捕捉数据集SPD并经归一化处理。结果表明,ELGAR在复杂快速交互场景下展现出强大生成能力,可推动动画、音乐教育、互动艺术创作等领域发展。
原文摘要 · Abstract (English)
The art of instrument performance stands as a vivid manifestation of human creativity and emotion. Nonetheless, generating instrument performance motions is a highly challenging task, as it requires not only capturing intricate movements but also reconstructing the complex dynamics of the performer-instrument interaction. While existing works primarily focus on modeling partial body motions, we propose Expressive ceLlo performance motion Generation for Audio Rendition (ELGAR), a state-of-the-art diffusion-based framework for whole-body fine-grained instrument performance motion generation solely from audio. To emphasize the interactive nature of the instrument performance, we introduce Hand Interactive Contact Loss (HICL) and Bow Interactive Contact Loss (BICL), which effectively guarantee the authenticity of the interplay. Moreover, to better evaluate whether the generated motions align with the semantic context of the music audio, we design novel metrics specifically for string instrument performance motion generation, including finger-contact distance, bow-string distance, and bowing score. Extensive evaluations and ablation studies are conducted to validate the efficacy of the proposed methods. In addition, we put forward a motion generation dataset SPD-GEN, collated and normalized from the MoCap dataset SPD. As demonstrated, ELGAR has shown great potential in generating instrument performance motions with complicated and fast interactions, which will promote further development in areas such as animation, music education, interactive art creation, etc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。