arXiv:2409.11952cs.RO2024-09被引 20

人机协作弹钢琴,机器人能实时听懂并配合人类演奏。

Human-Robot Cooperative Piano Playing with Learning-Based Real-Time Music Accompaniment

  • 用RNN根据人弹的旋律预测和弦,实现音乐即兴伴奏。
  • 93%准确率下,机器人可实时同步人类节奏完成合奏。
  • 通过熵值评估协作质量,适合人机艺术交互研究者。

机器学习的发展推动了音乐娱乐机器人的进步。然而,人机协同乐器演奏仍具挑战性,尤其在复杂的动作协调与时间同步方面。本文提出一种基于非语言线索的人机协作钢琴演奏理论框架:首先,设计了一种基于循环神经网络(RNN)的音乐即兴模型,根据人类的旋律输入预测合适的和弦进行;其次,提出一种行为自适应控制器,实现无缝的时间同步,使协作机器人生成和谐的伴奏声学效果。该协作系统考虑了人与机器人之间的双向信息流动。我们开发了一个基于熵的评估系统,用于分析不同通信模态在人机协作中的影响。实验表明,该RNN即兴模型可达到93%的准确率;同时,在使用模型预测控制(MPC)自适应控制器时,机器人可在同声部表演中实时响应人类队友,实现即时伴奏。所提出的框架已在艺术钢琴演奏任务中验证有效,显著提升了人机协同演奏的流畅性与表现力。

原文摘要 · Abstract (English)

Recent advances in machine learning have paved the way for the development of musical and entertainment robots. However, human-robot cooperative instrument playing remains a challenge, particularly due to the intricate motor coordination and temporal synchronization. In this paper, we propose a theoretical framework for human-robot cooperative piano playing based on non-verbal cues. First, we present a music improvisation model that employs a recurrent neural network (RNN) to predict appropriate chord progressions based on the human's melodic input. Second, we propose a behavior-adaptive controller to facilitate seamless temporal synchronization, allowing the cobot to generate harmonious acoustics. The collaboration takes into account the bidirectional information flow between the human and robot. We have developed an entropy-based system to assess the quality of cooperation by analyzing the impact of different communication modalities during human-robot collaboration. Experiments demonstrate that our RNN-based improvisation can achieve a 93\% accuracy rate. Meanwhile, with the MPC adaptive controller, the robot could respond to the human teammate in homophony performances with real-time accompaniment. Our designed framework has been validated to be effective in allowing humans and robots to work collaboratively in the artistic piano-playing task.

人机协作音乐生成实时控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。