用AI模型自动生成物理任务奖励,让仿真机器人学会更多动作。
GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill
- 用语言模型生成任务约束,视觉语言模型评估动作自然度,迭代优化奖励。
- 相比旧方法,动作自然度提升22.2%,任务完成率提高25.7%,训练快8.4倍。
- 适合想在仿真中快速训练通用物理技能的研究者和开发者。
在模拟环境中学习开放词汇的物理技能是人工智能的重要挑战。当前强化学习方法存在明显局限:人工设计奖励难以扩展到多样任务,基于示范的方法则难以超越训练分布。本文提出GROVE,一种无需人工工程或特定任务示范的通用奖励框架。核心思路是将大语言模型(LLMs)与视觉语言模型(VLMs)结合——LLMs生成精确的物理约束以捕捉任务需求,VLMs评估动作的语义与自然度。通过迭代优化,VLM反馈持续改进LLM生成的约束,形成自提升的奖励系统。为弥合仿真与真实图像间的域差距,我们设计轻量级映射器Pose2CLIP,无需复杂渲染即可将代理姿态直接投影至语义特征空间。在多种实体和学习范式下的实验表明,GROVE在运动自然度上提升22.2%,任务完成率提高25.7%,且训练速度加快8.4倍,为模拟环境中的可扩展物理技能获取奠定新基础。
原文摘要 · Abstract (English)
Learning open-vocabulary physical skills for simulated agents presents a significant challenge in artificial intelligence. Current reinforcement learning approaches face critical limitations: manually designed rewards lack scalability across diverse tasks, while demonstration-based methods struggle to generalize beyond their training distribution. We introduce GROVE, a generalized reward framework that enables open-vocabulary physical skill learning without manual engineering or task-specific demonstrations. Our key insight is that Large Language Models(LLMs) and Vision Language Models(VLMs) provide complementary guidance -- LLMs generate precise physical constraints capturing task requirements, while VLMs evaluate motion semantics and naturalness. Through an iterative design process, VLM-based feedback continuously refines LLM-generated constraints, creating a self-improving reward system. To bridge the domain gap between simulation and natural images, we develop Pose2CLIP, a lightweight mapper that efficiently projects agent poses directly into semantic feature space without computationally expensive rendering. Extensive experiments across diverse embodiments and learning paradigms demonstrate GROVE's effectiveness, achieving 22.2% higher motion naturalness and 25.7% better task completion scores while training 8.4x faster than previous methods. These results establish a new foundation for scalable physical skill acquisition in simulated environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。