arXiv:2603.16542cs.RO2026-03被引 2

让机器人学习更靠谱:按动作后果可信度重加权,避免低质量数据干扰。

Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting

  • 根据动作后结果的可解释性,动态分配训练样本权重
  • 在多个机器人数据集上提升策略适应性能,尤其在混合数据中表现更稳
  • 无需奖励函数,适配扩散与流匹配模型,适合真实场景数据

离线后训练通过监督回归将预训练机器人策略适配到目标数据集。但实际机器人数据异质性强:包含不同机械结构、摄像头配置及质量参差的示范轨迹,许多轨迹反映的是恢复行为、操作者技能不一致或弱信息监督。统一后训练对所有样本一视同仁,可能平均掉冲突或低可信度数据。我们提出后验-转移重加权(PTR),一种无奖励、保守的后训练方法,决定每个样本对更新的影响程度。对每个样本,将观察到的动作后果编码为潜在目标,插入候选目标池,用独立的转移评分器估计目标索引的softmax识别后验。后验与均匀分布的比值定义为PTR得分,经截断与混合后作为权重,通过自归一化加权回归应用于原始动作目标。该方法无需可计算的策略似然,兼容扩散与流匹配动作头。不再无差别信任所有记录监督,而根据当前表征下动作后果的可归因性重新分配信用,显著提升对异质机器人数据的保守离线适应能力。

原文摘要 · Abstract (English)

Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.

机器人学习离线强化学习策略优化数据重加权

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。