arXiv:2512.12842cs.ROcs.AI2025-12被引 3

通过结构化功能接地,让机器人在复杂环境中灵活完成多类任务。

SAGA: Open-World Mobile Manipulation via Structured Affordance Grounding

  • 用视觉中的功能热图抽象任务意图,分离语义与控制
  • 在11个真实任务中显著优于端到端和模块化基线
  • 支持语言、点选、示范等多种指令形式,零样本执行

我们提出SAGA,一种通用且自适应的视觉-运动控制框架,可跨环境、任务目标和用户需求实现泛化。核心思想是将高层语义意图与底层视觉运动控制解耦,通过基于功能的任务表征,在观察环境中显式地对齐任务目标。利用多模态基础模型,SAGA将任务表示映射为3D功能热图,突出任务相关实体,同时忽略干扰的外观变化。这些具身化的功能使我们能在多任务示范数据上训练条件策略,实现全身控制。在统一框架下,SAGA可处理语言指令、选定点及示例演示等多种输入形式,支持零样本执行与少样本适应。我们在四足机械臂上实现了SAGA,并在11个真实世界任务中进行了广泛实验,结果表明其性能显著优于端到端和模块化基线。这些结果共同证明,结构化功能接地为通用移动操作提供了一条可扩展且高效的技术路径。

原文摘要 · Abstract (English)

We present SAGA, a versatile and adaptive framework for visuomotor control that can generalize across various environments, task objectives, and user specifications. To efficiently learn such capability, our key idea is to disentangle high-level semantic intent from low-level visuomotor control by explicitly grounding task objectives in the observed environment. Using an affordance-based task representation, we express diverse and complex behaviors in a unified, structured form. By leveraging multimodal foundation models, SAGA grounds the proposed task representation to the robot's visual observation as 3D affordance heatmaps, highlighting task-relevant entities while abstracting away spurious appearance variations that would hinder generalization. These grounded affordances enable us to effectively train a conditional policy on multi-task demonstration data for whole-body control. In a unified framework, SAGA can solve tasks specified in different forms, including language instructions, selected points, and example demonstrations, enabling both zero-shot execution and few-shot adaptation. We instantiate SAGA on a quadrupedal manipulator and conduct extensive experiments across eleven real-world tasks. SAGA consistently outperforms end-to-end and modular baselines by substantial margins. Together, these results demonstrate that structured affordance grounding offers a scalable and effective pathway toward generalist mobile manipulation.

移动操作功能接地多模态通用控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。