用非语言动作触发服务机器人，突破15%响应瓶颈。
Breaking the 15% Barrier: A Real-World Data-Driven System for Proactive Social Robot Triggered by User Nonverbal Cues
- 基于视频实时识别多人多标签非语言动作
- 15.3%的机器人回应由非语言动作触发
- 无需手工规则，支持主动响应适合零售场景
零售店服务机器人常依赖语音流水线（STT-LLM-TTS），但许多互动由靠近、挥手、指物或展示物品等非语言行为发起。本研究在真实商店部署中使用远程操控人形机器人，发现15.3%的机器人回应由非语言输入触发，暴露出纯音频对话系统的局限性。基于对顾客行为的分析，定义一组高频且与服务相关的非语言线索，开发了实时多人群体、多标签视频识别系统。提出一种对话框架，将识别出的非语言线索作为令牌输入大模型生成回应，并在用户展示物品时可选融合视觉-语言模型，实现无规则的主动响应。离线评估验证了非语言触发回合的有效性，并上线原型系统实现实时响应。
原文摘要 · Abstract (English)
Service robots in retail stores increasingly rely on cascaded speech pipelines (STT-LLM-TTS), yet many customer-robot interactions are initiated or guided by nonverbal behaviors such as approaching, waving, pointing, or showing items. This paper studies such cues in a real-world store deployment with a teleoperated humanoid robot and shows that a non-negligible portion of robot turns are triggered by nonverbal behaviors rather than spoken input, revealing a limitation of audio-only dialogue systems. In a 6-day in-the-wild deployment, 15.3\% of robot utterances were initiated by users' nonverbal behaviors rather than spoken input. Based on an analysis of observed customer behaviors, we define a set of frequent, service-relevant nonverbal cues and develop a real-time multi-person, multi-label recognizer that runs online from video. We then propose a dialogue framework that conditions LLM-based utterance generation on recognized nonverbal cue tokens, and optionally leverages a vision-language model when items are shown, enabling proactive robot responses without hand-crafted rules. We evaluate the approach offline on nonverbal-triggered turns and demonstrate an online prototype that reacts to users' nonverbal cues in real time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。