arXiv:2606.26175cs.RO2026-06

用微任务分解+多视角语言模型奖励,让机器人更高效完成长时序操作。

RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards

论文配图:RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards
图 1 · 摘自论文原文
  • 将复杂任务拆解为多个语言描述的微任务,动态切换以细化学习信号
  • 相比单提示语言模型,奖励信号更丰富,使学习速度提升约3倍
  • 适合长时序机器人操作任务,尤其对初始条件随机的任务有效

机器人长时序操作中的强化学习常依赖人工设计密集奖励函数,难以调优且易失效;或通过人类演示/偏好学习奖励,成本高昂。近期工作使用预训练视觉-语言模型(VLM)作为零样本奖励模型,仅需一个文本提示即可替代,但针对随机初始状态的复杂任务,单一全局提示导致奖励信号在轨迹中大部分时间近乎平坦,阻碍早期进展感知。本文提出强化微任务学习(RMTL),将任务分解为若干语言描述的微任务,训练智能体在不同阶段动态切换。每步接收基于当前激活微任务提示的多视角VLM奖励,通过多摄像头平均降低视图遮挡影响。采用逆向课程逐步引入更难初始状态,并先用基于距离的固定规则训练策略,再替换为可学习的分层管理者,实现从规则驱动到端到端学习的过渡。在Fetch操作环境中,使用三个阶段特异性提示,无需额外提示调优。实验表明,相较单提示VLM奖励,RMTL提供更具信息量的奖励信号,显著加速学习过程。结果表明,将语言模型奖励按微任务细化,能大幅提升语言引导强化学习在机器人操作中的可扩展性。

原文摘要 · Abstract (English)

Reinforcement learning (RL) for robotic manipulation often requires manually designing a dense reward function, which is difficult to tune and often fragile, or learning a reward from human demonstrations or preferences, which can be expensive. A recent line of work uses pretrained vision-language models (VLMs) as zero-shot reward models, replacing these costs with a single text prompt. However, we argue that a single global prompt is too coarse for long-horizon manipulation tasks with randomized initial conditions. The single-prompt VLM reward is near-flat for much of the trajectory, making early progress hard for the agent to detect. We propose Reinforced Micro-Task Learning (RMTL), an approach that decomposes a manipulation task into a small set of language-described micro-tasks and trains the agent to switch between them. At each step, the agent receives a multi-view VLM reward computed using the prompt of the currently active micro-task and averaged across multiple camera views to reduce the effect of view-specific occlusions. A reverse curriculum gradually exposes the agent to harder initial conditions, while a PPO worker is first trained with a fixed distance-based rule that selects the active micro-task. We then replace this rule with a learned hierarchical manager, turning rule-based phase selection into a fully learned hierarchical policy. We instantiate RMTL on the Fetch manipulation environment using three short stage-specific prompts and without additional prompt tuning. Experiments show that RMTL provides more informative reward signals than single-prompt VLM rewards, enabling faster learning. These results suggest that decomposing VLM rewards into micro-task-specific language prompts can substantially improve the scalability of language-guided reinforcement learning for robotic manipulation.

机器人操作强化学习语言模型微任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。