让机器人通过对话澄清模糊指令,实现主动协作。
Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialogue
- 用视觉语言模型与扩散模型结合,边对话边执行任务。
- 在11个真实任务中表现优于现有方法,对话纠错能力显著。
- 适合研究具身智能、人机协作的开发者参考。
具身智能体是能在物理世界中感知、推理并行动的智能系统。尽管机器人领域长期致力于构建这类通用智能体,但当前基于视觉-语言-动作(VLA)模型的系统大多遵循僵化的“听命即执行”模式,假设指令无歧义且被动执行,无法通过对话消除不确定性。为此,我们提出Ask-to-Clarify——一个统一的端到端框架,将多轮消歧对话与底层视觉运动控制无缝集成,无需依赖高层动作原语或外部规划器。该框架融合基于视觉语言模型的认知规划器与基于扩散模型的运动执行器,并引入语义-视觉对齐适配器作为跨模态接口,实现语义意图与视觉感知流的协同融合。此外,我们发现微调视觉运动模块会引发严重灾难性遗忘,导致对话能力完全丢失。为此,我们提出两阶段知识隔离训练策略,有效解耦对话逻辑与物理操作。在11个真实任务上的大量评估表明,该框架显著优于现有方法,为构建真正协作的具身智能体提供了可行路径。
原文摘要 · Abstract (English)
Embodied agents are intelligent systems designed to perceive, reason, and act within the physical world. While the robotics community has long strived to build such versatile agents, a fundamental limitation persists: most current VLA-based models operate under a rigid ``Listen-and-Act'' paradigm. These systems assume instructions are unambiguous and execute them in a passive fashion, preventing them from resolving uncertainty through dialogue. To address this, we propose Ask-to-Clarify, a unified end-to-end framework that seamlessly integrates multi-turn disambiguation dialogue with low-level visuomotor control, eliminating the reliance on high-level action primitives or external planners. Specifically, Ask-to-Clarify synergizes a VLM-based Cognitive Planner with a Diffusion-based Motor Executor. To bridge the disparity between high-level disambiguation and low-level execution, we introduce a Semantic-Visual Alignment Adapter, which functions as a cross-modal interface to synthesize semantic intent with visual perceptual streams. Furthermore, we observe severe catastrophic forgetting: visuomotor fine-tuning completely erases dialogue capabilities. To overcome this, we propose a two-stage knowledge-insulation training strategy, effectively decoupling dialogue logic from physical manipulation. Extensive evaluations across 11 real-world tasks demonstrate that \framework{} significantly outperforms existing methods, offering a promising path toward building truly collaborative embodied agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。