arXiv:2505.02483cs.ROcs.AI2025-05ICRA被引 3

用大模型自动调度奖励,让机器人更高效学会复杂动作。

Automated Hybrid Reward Scheduling via Large Language Models for Robotic Skill Learning

  • 用大模型动态分配不同奖励的权重,分阶段优化学习过程。
  • 在多个高自由度任务中平均提升6.48%性能表现。
  • 适合需要精细动作控制的机器人学习场景。

让高自由度机器人学习特定技能面临挑战,因机械系统动力学复杂。强化学习(RL)是潜在解决方案,但需设计多个奖励函数以满足运动约束。现有方法通常无差别地累加所有奖励成分来优化价值函数和策略。我们认为这种统一包含所有奖励成分的方式效率低下,限制了机器人学习效果。为此,我们提出基于大语言模型(LLMs)的自动化混合奖励调度(AHRS)框架。该范式在策略优化过程中动态调整各奖励成分的学习强度,使机器人以渐进、结构化的方式掌握技能。具体地,设计一个多分支价值网络,每个分支对应一个独立的奖励成分。训练时,每个分支被赋予反映其重要性的权重,这些权重由大模型根据预设规则自动生成。大模型事先根据任务描述生成规则集,并在训练中通过语言提示评估各分支表现,从中选择合适的权重计算规则。实验结果表明,该方法在多个高自由度机器人任务中平均提升6.48%性能。

原文摘要 · Abstract (English)

Enabling a high-degree-of-freedom robot to learn specific skills is a challenging task due to the complexity of robotic dynamics. Reinforcement learning (RL) has emerged as a promising solution; however, addressing such problems requires the design of multiple reward functions to account for various constraints in robotic motion. Existing approaches typically sum all reward components indiscriminately to optimize the RL value function and policy. We argue that this uniform inclusion of all reward components in policy optimization is inefficient and limits the robot's learning performance. To address this, we propose an Automated Hybrid Reward Scheduling (AHRS) framework based on Large Language Models (LLMs). This paradigm dynamically adjusts the learning intensity of each reward component throughout the policy optimization process, enabling robots to acquire skills in a gradual and structured manner. Specifically, we design a multi-branch value network, where each branch corresponds to a distinct reward component. During policy optimization, each branch is assigned a weight that reflects its importance, and these weights are automatically computed based on rules designed by LLMs. The LLM generates a rule set in advance, derived from the task description, and during training, it selects a weight calculation rule from the library based on language prompts that evaluate the performance of each branch. Experimental results demonstrate that the AHRS method achieves an average 6.48% performance improvement across multiple high-degree-of-freedom robotic tasks.

机器人学习强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。