arXiv:2510.23763cs.ROcs.CL2025-10被引 8

机器人通过听对话、看环境主动理解意图,实现无需指令的协作操作。

RoboOmni: Proactive Robot Manipulation in Omni-modal Context

  • 融合视听信号,从对话和环境音中推断用户意图
  • 在仿真与真实场景中成功率更高,响应速度更快
  • 适合需要主动服务的智能机器人应用

多模态大语言模型的进展推动了视觉-语言-动作模型在机器人操作中的发展。然而,现有方法主要依赖明确指令,而现实交互中人类很少直接下达命令。有效协作需机器人主动推断意图。本文提出跨模态上下文指令新范式,即从口语对话、环境声音和视觉线索中推断意图,而非显式指令。为此,我们构建RoboOmni——一个基于端到端全模态大模型的感知-思考-说话-执行框架,统一意图识别、交互确认与动作执行。RoboOmni通过时空融合音频与视觉信号实现鲁棒意图识别,并支持直接语音交互。针对主动意图识别缺乏训练数据的问题,我们构建了包含140万条任务实例、5000+说话人、2400种事件音效、640种背景音及六类情境指令的OmniAction数据集。仿真与真实场景实验表明,RoboOmni在成功率、推理速度、意图识别准确率及主动协助能力上均优于基于文本和语音识别的基线方法。

原文摘要 · Abstract (English)

Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.

机器人操作多模态模型主动协作意图理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。