让机器人学习更靠谱:按动作后果可信度重加权,避免低质量数据干扰。
Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting
- 根据动作后结果的可解释性,动态分配训练样本权重
- 在多个机器人数据集上提升策略适应性能,尤其在混合数据中表现更稳
- 无需奖励函数,适配扩散与流匹配模型,适合真实场景数据
离线后训练通过监督回归将预训练机器人策略适配到目标数据集。但实际机器人数据异质性强:包含不同机械结构、摄像头配置及质量参差的示范轨迹,许多轨迹反映的是恢复行为、操作者技能不一致或弱信息监督。统一后训练对所有样本一视同仁,可能平均掉冲突或低可信度数据。我们提出后验-转移重加权(PTR),一种无奖励、保守的后训练方法,决定每个样本对更新的影响程度。对每个样本,将观察到的动作后果编码为潜在目标,插入候选目标池,用独立的转移评分器估计目标索引的softmax识别后验。后验与均匀分布的比值定义为PTR得分,经截断与混合后作为权重,通过自归一化加权回归应用于原始动作目标。该方法无需可计算的策略似然,兼容扩散与流匹配动作头。不再无差别信任所有记录监督,而根据当前表征下动作后果的可归因性重新分配信用,显著提升对异质机器人数据的保守离线适应能力。
原文摘要 · Abstract (English)
Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。