让机器人像人一样用多种感官完成复杂操作,成功率超80%。
MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation

- 根据任务指令动态选择传感器并生成感知描述
- 将不同传感器数据转为统一图像表示,提升融合效率
- 零样本泛化能力强,适合需要多模态感知的机器人场景
人类在与物理世界交互时会自然使用多种感知模态,而现有用于机器人的视觉-语言-动作(VLA)模型大多仅依赖RGB观测,难以获取温度、声音或雷达响应等物理属性。本文提出MuseVLA,一种自适应多模态感知的VLA模型,可按需调用新传感器作为工具。给定任务指令和视觉上下文,MuseVLA首先生成传感器标记与目标描述,类似工具调用;随后将选定传感器测量结果转化为具身化的传感器图像,作为异构数据融合与动作生成的统一中间表示。该设计将传感器特异性处理与VLA主干解耦,实现多种模态高效集成。为减少对昂贵多感官机器人数据集的依赖,我们进一步提出数据合成管道,通过在现有RGB视频数据集上添加具身传感器图像,实现对未见传感任务的泛化。我们在真实机器人上评估MuseVLA,涵盖需多模态输入的精细手部操作任务,包括温度引导的抓取放置、音频驱动的物体搜寻及雷达辅助的隐藏物体检索。MuseVLA平均成功率达80.6%,显著优于仅用RGB和多感官基线模型,并展现出强零样本能力。代码、模型与数据集已开源。
原文摘要 · Abstract (English)
Humans naturally leverage diverse sensing modalities to interact with the physical world, while most Vision-Language-Action (VLA) models for robotics rely solely on RGB observations. This limits their ability to perceive physical properties that are difficult or impossible to infer from RGB cameras, such as temperature, sound, or radar response. We present MuseVLA, an adaptive multimodal sensing VLA model that integrates novel sensors as on-demand tools for robotic manipulation. Given a task instruction and visual context, MuseVLA first generates a sensor token and target description that select the sensing modality to invoke and what to attend to, analogous to a tool call with arguments. It then converts the selected sensor measurement into a grounded sensor image, a unified intermediate representation that encodes heterogeneous readings for multimodal fusion and action generation. This design decouples sensor-specific processing from the VLA backbone, enabling efficient integration of diverse modalities. To reduce the need for expensive multisensory robot datasets, we further introduce a data synthesis pipeline that augments existing RGB video datasets with grounded sensor images, enabling generalization to unseen sensor-guided tasks. We evaluate MuseVLA on a real-world robot across challenging dexterous hand manipulation tasks that require multimodal sensing inputs, including temperature-guided pick-and-place, audio-driven object search, and radar-assisted hidden object retrieval. MuseVLA achieves 80.6% success rate on average, outperforming RGB-only and multisensory VLA baselines significantly, and exhibits strong zero-shot capabilities on unseen tasks. Code, model and dataset are available at https://github.com/microsoft/MuseVLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。