用大模型把少量人类示范扩展成海量机器人数据，自动生成可执行动作。
LLM Trainer: Automated Robotic Data Generation via Demonstration Augmentation using LLMs
- 用大模型分析示范视频，提取关键帧和物体关系
- 仅需1次示范，生成成功率超90%的可执行新轨迹
- 适合机器人学习、自动化数据生成研究者
我们提出LLM Trainer，一个完全自动化的流水线，利用大语言模型（LLMs）的世界知识，将极少量人类示范（最少仅1个）转化为用于模仿学习的大规模机器人数据集。该方法分两步：(1) 离线示范标注，提取关键帧、显著物体及姿态-物体关系；(2) 在线关键姿态重定向，根据初始观察将这些关键帧适配到新场景。基于修正的关键点，系统对原始示范进行形变生成新轨迹，并执行；若成功则保存为新示范。由于标注可跨场景复用，我们采用Thompson采样优化标注过程，显著提升生成成功率。我们在多种任务上评估该方法，结果表明其数据标注性能持续优于人工设计基线。我们进一步提出一种融合优化后的LLM前馈规划与学习型反馈模仿控制器的集成策略。最后，我们在Franka Emika Panda机器人上验证了硬件可行性。
原文摘要 · Abstract (English)
We present LLM Trainer, a fully automated pipeline that leverages the world knowledge of Large Language Models (LLMs) to transform a small number of human demonstrations (as few as one) into a large robot dataset for imitation learning. Our approach decomposes demonstration generation into two steps: (1) offline demonstration annotation that extracts keyframes, salient objects, and pose-object relations; and (2) online keypose retargeting that adapts those keyframes to a new scene, given an initial observation. Using these modified keypoints, our system warps the original demonstration to generate a new trajectory, which is then executed, and the resulting demo, if successful, is saved. Because the annotation is reusable across scenes, we use Thompson sampling to optimize the annotation, significantly improving generation success rate. We evaluate our method on a range of tasks, and find that our data annotation method consistently outperforms expert-engineered baselines. We further show an ensemble policy that combines the optimized LLM feed-forward plan with a learned feedback imitation learning controller. Finally, we demonstrate hardware feasibility on a Franka Emika Panda robot. For additional materials and demonstration videos, please see the project website: https://sites.google.com/andrew.cmu.edu/llm-trainer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。