让机器人说话、表情和动作同步，还能在本地运行。
Bridging Speech, Emotion, and Motion: a VLM-based Multimodal Edge-deployable Framework for Humanoid Robots
- 用视觉语言模型统一控制语音、表情与手势
- 边缘部署版本性能保持95%以上，支持离线运行
- 适合需要自然互动的教育、服务类机器人
高效人机交互需要情感丰富的多模态表达，但多数人形机器人缺乏语音、面部表情与手势的协调。同时，实际部署要求设备端解决方案，能在无持续云端连接下自主运行。为弥合语音（Speech）、情感（Emotion）与运动（Motion）的鸿沟，我们提出基于视觉语言模型的框架SeM²，通过三个关键组件实现情感一致的多模态交互：捕捉用户上下文线索的多模态感知模块、用于响应规划的思维链推理，以及确保言语内容与肢体表达精准时序对齐的新机制SSAM。我们实现了云端与边缘部署版本（SeM²_e），后者经知识蒸馏可在边缘硬件高效运行，仍保持相对性能的95%。综合评估表明，该方法在自然度、情感清晰度和模态一致性上显著优于单模态基线，推动了适用于多样化现实环境的社会化人形机器人发展。
原文摘要 · Abstract (English)
Effective human-robot interaction requires emotionally rich multimodal expressions, yet most humanoid robots lack coordinated speech, facial expressions, and gestures. Meanwhile, real-world deployment demands on-device solutions that can operate autonomously without continuous cloud connectivity. To bridging \underline{\textit{S}}peech, \underline{\textit{E}}motion, and \underline{\textit{M}}otion, we present \textit{SeM$^2$}, a Vision Language Model-based framework that orchestrates emotionally coherent multimodal interactions through three key components: a multimodal perception module capturing user contextual cues, a Chain-of-Thought reasoning for response planning, and a novel Semantic-Sequence Aligning Mechanism (SSAM) that ensures precise temporal coordination between verbal content and physical expressions. We implement both cloud-based and \underline{\textit{e}}dge-deployed versions (\textit{SeM$^2_e$}), with the latter knowledge distilled to operate efficiently on edge hardware while maintaining 95\% of the relative performance. Comprehensive evaluations demonstrate that our approach significantly outperforms unimodal baselines in naturalness, emotional clarity, and modal coherence, advancing socially expressive humanoid robotics for diverse real-world environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。