用自然语言生成机器人示范视频，让机械臂高效学会新技能。
Learning Novel Skills from Language-Generated Demonstrations
- 用视觉语言模型和视频扩散模型生成新技能的演示视频
- 在新任务上技能达成率提升至原来的三倍
- 适合想快速训练机器人新动作的研究者和工程师
机器人在多样场景中需掌握新技能，但现有学习方法依赖人工示范或环境交互，成本高且有安全风险。本文提出DemoGen框架，通过自然语言指令生成高质量示范视频，使机器人能有效学习新技能。在MetaWorld仿真环境中，该方法生成的演示视频具有高保真度和可靠性。利用这些生成示范，多种技能学习算法在新任务上的完成率提升至原始水平的三倍。结果表明，该方法为机器人直观、智能地获取新技能提供了新路径。
原文摘要 · Abstract (English)
Robots are increasingly deployed across diverse domains to tackle tasks requiring novel skills. However, current robot learning algorithms for acquiring novel skills often rely on demonstration datasets or environment interactions, resulting in high labor costs and potential safety risks. To address these challenges, this study proposes DemoGen, a skill-learning framework that enables robots to acquire novel skills from natural language instructions. DemoGen leverages the vision-language model and the video diffusion model to generate demonstration videos of novel skills, which enabling robots to learn new skills effectively. Experimental evaluations in the MetaWorld simulation environments demonstrate the pipeline's capability to generate high-fidelity and reliable demonstrations. Using the generated demonstrations, various skill learning algorithms achieve an accomplishment rate three times the original on novel tasks. These results highlight a novel approach to robot learning, offering a foundation for the intuitive and intelligent acquisition of novel robotic skills. (Project website: https://aoqunjin.github.io/LNSLGD/)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。