用提示学习让大模型直接预测机器人动作，无需训练。
In-Context Learning Enables Robot Action Prediction in LLMs
- 从视频中提取关键帧，转为文本描述作为提示。
- 在模拟和真实场景中优于零样本和基线方法。
- 适合想快速部署大模型做机器人控制的研究者。
最近，大语言模型（LLMs）在自然语言领域通过上下文学习（ICL）取得了显著成功。然而，利用 LLM 的 ICL 能力直接预测机器人动作仍鲜有研究。本文提出 RoboPrompt 框架，使现成的纯文本大模型无需训练即可通过 ICL 直接预测机器人动作。方法首先启发式地识别包含重要时刻的视频关键帧；接着从这些关键帧中提取末端执行器动作及物体初始位姿，并转换为文本描述；最后构建结构化模板，将文本描述与任务指令组合成 ICL 示例。测试时，大模型可直接据此预测动作。大量实验表明，RoboPrompt 在模拟和真实场景中均优于零样本及 ICL 基线方法。
原文摘要 · Abstract (English)
Recently, Large Language Models (LLMs) have achieved remarkable success using in-context learning (ICL) in the language domain. However, leveraging the ICL capabilities within LLMs to directly predict robot actions remains largely unexplored. In this paper, we introduce RoboPrompt, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions through ICL without training. Our approach first heuristically identifies keyframes that capture important moments from an episode. Next, we extract end-effector actions from these keyframes as well as the estimated initial object poses, and both are converted into textual descriptions. Finally, we construct a structured template to form ICL demonstrations from these textual descriptions and a task instruction. This enables an LLM to directly predict robot actions at test time. Through extensive experiments and analysis, RoboPrompt shows stronger performance over zero-shot and ICL baselines in simulated and real-world settings. Our project page is available at https://davidyyd.github.io/roboprompt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。