无需动作捕捉,直接将乐谱转为机器人演奏动作。
From Score to Sound: An End-to-End MIDI-to-Motion Pipeline for Robotic Cello Performance
- 端到端将MIDI乐谱转为带碰撞规避的运弓动作。
- 132人参与音乐图灵测试,表现接近人类演奏水平。
- 公开5首曲目机器人演奏数据,支持后续研究。
机器人演奏乐器需精准控制音准、音色与音乐表现力。弦乐器如大提琴对运弓角度和压力的控制要求极高,现有机器人演奏系统多依赖昂贵的动作捕捉技术,且无法实现类人视奏。本文提出一种端到端的MIDI到机器人运动的转换管道,利用UR5e机器人的Freedrive功能,直接将乐谱生成具备碰撞规避能力的运弓动作,实现类人发声效果,无需动作捕捉。通过实时数据交换(RTDE)记录演奏时的关节数据,收集了五首标准曲目的机器人演奏数据并公开共享。为验证性能,设计音乐图灵测试,132名参与者对比评估机器人与人类演奏表现,结果建立首个机器人大提琴演奏基准。此外,提出残差强化学习方法优化基础控制策略,未来可用于提升换弦效率与音质。
原文摘要 · Abstract (English)
Robot musicians require precise control to obtain proper note accuracy, sound quality, and musical expression. Performance of string instruments, such as violin and cello, presents a significant challenge due to the precise control required over bow angle and pressure to produce the desired sound. While prior robotic cellists focus on accurate bowing trajectories, these works often rely on expensive motion capture techniques, and fail to sightread music in a human-like way. We propose a novel end-to-end MIDI score to robotic motion pipeline which converts musical input directly into collision-aware bowing motions for a UR5e robot cellist. Through use of Universal Robot Freedrive feature, our robotic musician can achieve human-like sound without the need for motion capture. Additionally, this work records live joint data via Real-Time Data Exchange (RTDE) as the robot plays, providing labeled robotic playing data from a collection of five standard pieces to the research community. To demonstrate the effectiveness of our method in comparison to human performers, we introduce the Musical Turing Test, in which a collection of 132 human participants evaluate our robot's performance against a human baseline. Human reference recordings are also released, enabling direct comparison for future studies. This evaluation technique establishes the first benchmark for robotic cello performance. Finally, we outline a residual reinforcement learning methodology to improve upon baseline robotic controls, highlighting future opportunities for improved string-crossing efficiency and sound quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。