让机器人学会手语，帮助聋哑人群体沟通
SignBot: Learning Human-to-Humanoid Sign Language Interaction
- 用仿脑结构控制机器人手语动作，实现自然拟态
- 支持多机器人平台和多种手语数据集，交互效果好
- 适合无障碍通信、人形机器人研发者使用
手语是一种以肢体动作和表情传递意义的自然视觉语言，是听障或重听人士（DHH）的重要沟通方式。然而，掌握手语的人数仍然有限，亟需技术手段弥合沟通鸿沟。基于近期具身人形机器人的进展，本文提出SignBot，一种面向人机手语交互的新框架。SignBot融合小脑启发的动作控制模块与大脑导向的语义理解模块，包含：1）运动重定向，将人类手语数据转化为机器人可执行的运动学参数；2）运动控制，采用学习驱动范式构建稳健的人形机器人手语追踪策略；3）生成式交互，集成手语翻译器、应答器与生成器，实现人机间自然有效的手语交流。仿真与真实实验表明，SignBot能有效促进人机互动，并在多种机器人平台与手语数据集上完成手语动作。该工作推动了具身人形机器人在自动手语交互中的应用，为提升听障群体沟通可及性提供可行方案。
原文摘要 · Abstract (English)
Sign language is a natural and visual form of language that uses movements and expressions to convey meaning, serving as a crucial means of communication for individuals who are deaf or hard-of-hearing (DHH). However, the number of people proficient in sign language remains limited, highlighting the need for technological advancements to bridge communication gaps and foster interactions with minorities. Based on recent advancements in embodied humanoid robots, we propose SignBot, a novel framework for human-robot sign language interaction. SignBot integrates a cerebellum-inspired motion control component and a cerebral-oriented module for comprehension and interaction. Specifically, SignBot consists of: 1) Motion Retargeting, which converts human sign language datasets into robot-compatible kinematics; 2) Motion Control, which leverages a learning-based paradigm to develop a robust humanoid control policy for tracking sign language gestures; and 3) Generative Interaction, which incorporates translator, responser, and generator of sign language, thereby enabling natural and effective communication between robots and humans. Simulation and real-world experimental results demonstrate that SignBot can effectively facilitate human-robot interaction and perform sign language motions with diverse robots and datasets. SignBot represents a significant advancement in automatic sign language interaction on embodied humanoid robot platforms, providing a promising solution to improve communication accessibility for the DHH community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。