不训练模型，用大模型指挥智能体完成开放世界任务。
TANGO: Training-free Embodied AI Agents for Open-world Tasks
- 用大模型编排基础导航和记忆探索模块，实现零样本任务组合。
- 在三种开放世界任务中达到最优零样本性能，无需微调。
- 适合研究零样本智能体、少样本泛化与具身认知的学者。
大型语言模型（LLMs）已展现出将多个模块组合成程序以执行复杂图像推理任务的能力。本文提出TANGO，一种将此类程序组合能力扩展至具身智能体的方法，使智能体能够在真实环境中观察并行动。具体而言，通过结合简单的点目标导航模型与基于记忆的探索策略作为基础原语，我们证明单一模型可在不进行额外训练的情况下应对多种任务。让大模型仅通过提示中的少量示例，即可编排这些基础原语来解决特定任务。我们在三个关键具身AI任务上评估该方法：开放集物体目标导航、多模态终身导航与开放具身问答，在挑战性的零样本场景中实现最先进性能，且无需任何特定微调。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated excellent capabilities in composing various modules together to create programs that can perform complex reasoning tasks on images. In this paper, we propose TANGO, an approach that extends the program composition via LLMs already observed for images, aiming to integrate those capabilities into embodied agents capable of observing and acting in the world. Specifically, by employing a simple PointGoal Navigation model combined with a memory-based exploration policy as a foundational primitive for guiding an agent through the world, we show how a single model can address diverse tasks without additional training. We task an LLM with composing the provided primitives to solve a specific task, using only a few in-context examples in the prompt. We evaluate our approach on three key Embodied AI tasks: Open-Set ObjectGoal Navigation, Multi-Modal Lifelong Navigation, and Open Embodied Question Answering, achieving state-of-the-art results without any specific fine-tuning in challenging zero-shot scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。