提出新评估指标CI-MSE,让离线验证更可靠。
Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

- 只计算任务关键段的误差,结合动作对齐提升预测精度
- 在多种检查点上相关性达-0.87,显著优于原始MSE的-0.61
- 适合需要快速迭代机器人策略的研究者使用
真实世界评估是机器人策略的金标准,但成本高、难复现且数据稀疏,难以有效比较相近模型。常用代理指标——专家示范上的验证损失,与真实表现相关性差。本文提出一种直观而有效的离线验证指标:关键区间均方误差(CI-MSE)。该方法仅在任务关键片段计算误差,并结合简单的动作对齐策略,更好匹配实际运行行为。在仿真和真实场景中,CI-MSE相比原始MSE显著提升了验证误差与实际表现的相关性。在大量策略检查点上,其斯皮尔曼等级相关系数达-0.87,远接近理想值-1,优于原始MSE的-0.61。敏感性分析显示该指标对超参数鲁棒。进一步研究了分布偏移下的有效性,提出了使用边界建议。本工作为加速机器人策略迭代提供了简单可靠的离线验证工具。
原文摘要 · Abstract (English)
Real-world evaluation is the gold standard for robot policies because it tests them against the physical conditions and deployment challenges they are ultimately designed to handle. However, real-world evaluation is also the bottleneck for iterating on robot policies: it is costly, difficult to reproduce, and often too sparse to reliably compare nearby model variants. A straightforward proxy for performance is validation loss on expert demonstrations, but this proxy is often poorly correlated with real-world performance. In this paper, we introduce Critical Interval MSE (CI-MSE), an intuitively simple yet effective offline validation metric. CI-MSE restricts error computation to task-critical segments and pairs it with simple action-alignment procedures that better match rollout-time behavior. Across simulation and real-world experiments, CI-MSE yields a stronger correlation between validation error and rollout performance than raw MSE. Across a wide range of policy checkpoints, CI-MSE achieves a Spearman's rank correlation of $-0.87$, much closer to the ideal value of $-1$ than raw MSE's $-0.61$, demonstrating a significant improvement. We show through sensitivity analysis that our metric is robust to a wide range of hyperparameters. We further study the effectiveness of CI-MSE under evaluation distribution shifts and suggest design boundaries when using this metric. In summary, this paper provides a simple and reliable offline validation tool for accelerating policy iteration. Project webpage: https://ci-mse.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。