arXiv:2606.17043cs.ROcs.LG2026-06

针对稀疏反馈的在线强化学习,提出分层优势加权方法提升机器人操作成功率。

Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes

论文配图:Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
图 1 · 摘自论文原文
  • 分离训练可行性和效率两个目标的评估头,用状态自适应门控融合信号。
  • 在三个真实机器人任务中将成功率从12%-44%提升至38%-92%。
  • 适合需要高精度动作调整的具身智能任务,如双臂复杂抓取。

当预训练的视觉-语言-动作(VLA)策略通过在线强化学习微调时,每个回放轨迹仅产生单一二元结果(成功或失败),但智能体更新需要逐步监督。现有方法通常将稀疏结果简化为单个标量奖励或优势信号,混淆了不同层次的反馈信息,并在基本任务成功后失去优化引导。首先,单一标量信号混淆了可行性与效率的目标;一旦达成基础成功,二元标签无法提供梯度以区分高效与低效完成。其次,实际回放包含自主执行和人工干预段落;若在边界处直接分配整个轨迹结果,会导致错误信用分配。为此,本文提出分层优势加权行为克隆(HABC),在不同数据子集上分别训练可行性与效率的评判头,并通过状态自适应门控 $g_t$ 融合其一步优势。该门控在成功不确定时优先考虑可行性,在可行性较高时转向效率;并将融合结果转化为对智能体损失函数的逐步权重。此外,引入干预感知信用分配,仅将结果标签限制于当前策略执行的段落,防止跨干预边界泄露监督信号。在三个接触密集型双臂任务的真实机器人实验中,HABC 将成功率从监督微调(SFT)基线的36%、44%、12% 提升至92%、88%、38%。

原文摘要 · Abstract (English)

When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage signal, which conflates distinct forms of transition-level feedback and provides limited guidance once basic task success becomes achievable. First, a single scalar signal conflates the two objectives of viability and efficiency; once basic success is achieved, the binary label provides no gradient to distinguish efficient completions from slow ones. Second, real-world rollouts mix autonomous and intervention segments; naively assigning episode outcomes across these boundaries introduces incorrect credit assignment. To address these issues, we propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for these two objectives on different data subsets and combines their outputs with a state-adaptive balance. A state-adaptive gate $g_t$ merges their one-step advantages, prioritizing viability when success is uncertain and shifting to efficiency only when viability is high, and converts the result into per-transition weights on the actor loss. Intervention-aware credit assignment further restricts outcome labels to segments executed by the current policy, preventing supervision from leaking across intervention boundaries. In real-robot experiments on three contact-rich bimanual tasks, HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.

在线强化学习机器人控制分层优化稀疏反馈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。