对比三种大模型整合方式，提升机器人理解指令与操作能力。
From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems
- 用端到端或模块化流程整合视觉语言模型，实现指令到动作的映射。
- 在零样本和少样本下验证，不同策略在泛化与数据效率间存在权衡。
- 适合研究具身智能、机器人控制及大模型应用的开发者参考。
基础模型（FMs）正被越来越多地用于连接具身智能体的语言与动作，但不同集成策略的操作特性仍缺乏深入探索，尤其是在复杂指令理解和动态环境中的多样化动作生成方面。本文通过两个案例研究，考察三种构建机器人系统的方法：端到端视觉-语言-动作（VLA）模型，以及结合视觉语言模型（VLMs）或多模态大语言模型（LLMs）的模块化流水线。实验涵盖复杂指令定位任务（评估细粒度指令理解与跨模态消歧）和物体操作任务（通过VLA微调实现技能迁移），在零样本和少样本设置下揭示了各方法在泛化能力与数据效率间的权衡。通过分析性能极限，提炼出面向语言驱动物理智能体的设计启示，并指出未来在真实环境中实现FM赋能机器人所面临的挑战与机遇。
原文摘要 · Abstract (English)
Foundation models (FMs) are increasingly used to bridge language and action in embodied agents, yet the operational characteristics of different FM integration strategies remain under-explored -- particularly for complex instruction following and versatile action generation in changing environments. This paper examines three paradigms for building robotic systems: end-to-end vision-language-action (VLA) models that implicitly integrate perception and planning, and modular pipelines incorporating either vision-language models (VLMs) or multimodal large language models (LLMs). We evaluate these paradigms through two focused case studies: a complex instruction grounding task assessing fine-grained instruction understanding and cross-modal disambiguation, and an object manipulation task targeting skill transfer via VLA finetuning. Our experiments in zero-shot and few-shot settings reveal trade-offs in generalization and data efficiency. By exploring performance limits, we distill design implications for developing language-driven physical agents and outline emerging challenges and opportunities for FM-powered robotics in real-world conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。