用视频直接生成机器人行为奖励函数,实现精准可控的运动学习。
Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning
- 通过视频转关键点轨迹,再由大模型生成奖励函数。
- 在双足和四足任务中,人类评分超越现有方法37.6%以上。
- 换视频即可快速学习走路、跑步等多样动作,适合快速原型开发。
腿式机器人行为学习因固有的不稳定性与复杂约束而面临重大挑战。近期研究尝试利用大语言模型(LLM)生成强化学习中的奖励函数,以替代专家手动设计。然而,仅依赖文本描述定义学习目标的方法难以实现方向明确、控制精确的行为学习。本文提出一种新的Video2Reward方法,直接从待模仿行为的视频中生成奖励函数。首先,通过视频转文本转换模块将目标行为视频中个体的运动信息转化为关键点轨迹坐标;随后,将轨迹输入大模型生成奖励函数,并用于策略训练。为提升奖励函数质量,我们设计了一种视频辅助的迭代奖励优化方案,通过视觉评估学习到的行为并生成文本反馈,引导大模型持续优化奖励函数,从而加速行为学习。在双足与四足机器人运动控制任务上的实验表明,该方法在人类归一化得分上优于当前最先进的基于LLM的奖励生成方法超过37.6%。更重要的是,仅通过更换视频输入,即可快速学习步行、奔跑等多种运动行为。
原文摘要 · Abstract (English)
Learning behavior in legged robots presents a significant challenge due to its inherent instability and complex constraints. Recent research has proposed the use of a large language model (LLM) to generate reward functions in reinforcement learning, thereby replacing the need for manually designed rewards by experts. However, this approach, which relies on textual descriptions to define learning objectives, fails to achieve controllable and precise behavior learning with clear directionality. In this paper, we introduce a new video2reward method, which directly generates reward functions from videos depicting the behaviors to be mimicked and learned. Specifically, we first process videos containing the target behaviors, converting the motion information of individuals in the videos into keypoint trajectories represented as coordinates through a video2text transforming module. These trajectories are then fed into an LLM to generate the reward function, which in turn is used to train the policy. To enhance the quality of the reward function, we develop a video-assisted iterative reward refinement scheme that visually assesses the learned behaviors and provides textual feedback to the LLM. This feedback guides the LLM to continually refine the reward function, ultimately facilitating more efficient behavior learning. Experimental results on tasks involving bipedal and quadrupedal robot motion control demonstrate that our method surpasses the performance of state-of-the-art LLM-based reward generation methods by over 37.6% in terms of human normalized score. More importantly, by switching video inputs, we find our method can rapidly learn diverse motion behaviors such as walking and running.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。