用语音和手势融合提升人机对话鲁棒性
TransforMerger: Transformer-based Voice-Gesture Fusion for Robust Human-Robot Communication
- 基于Transformer融合语音与手势输入，生成结构化指令
- 在噪声、错位数据下仍保持高准确率，优于传统方法
- 适合需要上下文理解的复杂人机协作场景
随着人机协作的发展，自然灵活的通信方式对有效机器人控制至关重要。传统单模态或规则驱动方法在面对噪声数据、信号错位及不匹配预设名称的对象描述（如‘拿那个红色物体’）时表现不佳。我们提出TransforMerger，一种基于Transformer的推理模型，通过融合语音与手势输入，推断出用于机器人操作的结构化动作指令。该方法将多模态信息合并为单一统一语句，再由语言模型处理。我们采用概率嵌入应对不确定性，并结合上下文场景理解解决模糊指代问题（如手势指向多个物体或‘这个’等模糊口语）。在模拟与真实实验中验证了其对噪声、错位和缺失信息的鲁棒性。结果表明，相较于确定性基线，TransforMerger在需更多上下文知识的场景中表现更优，显著提升了人机通信的鲁棒性与灵活性。代码与数据集见：http://imitrob.ciirc.cvut.cz/publications/transformerger。
原文摘要 · Abstract (English)
As human-robot collaboration advances, natural and flexible communication methods are essential for effective robot control. Traditional methods relying on a single modality or rigid rules struggle with noisy or misaligned data as well as with object descriptions that do not perfectly fit the predefined object names (e.g. 'Pick that red object'). We introduce TransforMerger, a transformer-based reasoning model that infers a structured action command for robotic manipulation based on fused voice and gesture inputs. Our approach merges multimodal data into a single unified sentence, which is then processed by the language model. We employ probabilistic embeddings to handle uncertainty and we integrate contextual scene understanding to resolve ambiguous references (e.g., gestures pointing to multiple objects or vague verbal cues like "this"). We evaluate TransforMerger in simulated and real-world experiments, demonstrating its robustness to noise, misalignment, and missing information. Our results show that TransforMerger outperforms deterministic baselines, especially in scenarios requiring more contextual knowledge, enabling more robust and flexible human-robot communication. Code and datasets are available at: http://imitrob.ciirc.cvut.cz/publications/transformerger.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。