arXiv:2512.17183cs.RO2025-12

让机器人听懂话后自然比手势,实时精准执行。

Semantic Co-Speech Gesture Synthesis and Real-Time Control for Humanoid Robots

  • 用大模型+自回归模型从语音生成有语义的手势动作
  • 真实机器人在20秒内完成复杂手势且保持平衡
  • 适合人机交互、服务机器人等需要自然表达的场景

我们提出一种端到端框架,实现语义相关的口部同步手势生成,并在人形机器人上实现实时部署。该系统通过融合先进的手势生成技术与稳健的物理控制策略,解决机器人自然非语言交流难题。核心创新在于引入语义感知的手势生成模块,基于大语言模型(LLMs)和自回归运动生成模型(Motion-GPT),从语音输入中提取具有表现力的参考动作。同时采用高保真模仿学习控制策略(MotionTracker),使Unitree G1人形机器人能够动态执行复杂动作并维持平衡。为确保可行性,使用鲁棒的通用运动重定向(GMR)方法弥合人体运动数据与机器人平台间的差异。综合评估表明,该系统生成的语义恰当、节奏协调的手势能被物理机器人准确跟踪与执行。据我们所知,这是首个实现自动、语义感知、口部同步手势生成并同步实现实时物理部署的完整方案,迈向通用现实应用的重要一步。

原文摘要 · Abstract (English)

We present an innovative end-to-end framework for synthesizing semantically meaningful co-speech gestures and deploying them in real-time on a humanoid robot. This system addresses the challenge of creating natural, expressive non-verbal communication for robots by integrating advanced gesture generation techniques with robust physical control. Our core innovation lies in the meticulous integration of a semantics-aware gesture synthesis module, which derives expressive reference motions from speech input by leveraging a generative retrieval mechanism based on large language models (LLMs) and an autoregressive Motion-GPT model. This is coupled with a high-fidelity imitation learning control policy, the MotionTracker, which enables the Unitree G1 humanoid robot to execute these complex motions dynamically and maintain balance. To ensure feasibility, we employ a robust General Motion Retargeting (GMR) method to bridge the embodiment gap between human motion data and the robot platform. Through comprehensive evaluation, we demonstrate that our combined system produces semantically appropriate and rhythmically coherent gestures that are accurately tracked and executed by the physical robot. To our knowledge, this work represents a significant step toward general real-world use by providing a complete pipeline for automatic, semantic-aware, co-speech gesture generation and synchronized real-time physical deployment on a humanoid robot.

手势生成人形机器人语音驱动实时控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。