提出机器人学习评估新标准，让实验结果更可信。
Robot Learning as an Empirical Science: Best Practices for Policy Evaluation
- 明确报告实验条件与成功标准，避免信息缺失
- 补充成功率外的多维度评估指标，提升分析深度
- 强调失败模式的定性描述，适合方法对比研究者参考
近年来机器人学习领域进展显著，新架构和新能力不断涌现；然而，文献中普遍以成功率（即成功运行占比）作为主要评估指标，尤其在物理实验中。多数论文仅报告成功率数值，缺乏对运行次数、初始条件、成功标准的说明，也缺少对行为模式和失败情况的定性描述，以及统计分析。本文主张，为推动领域发展，研究人员应提供更细致的方法评估，特别是针对物理机器人上学习策略的评估。为此，我们提出一系列最佳实践：明确报告实验设置，采用互补的成功率指标，开展统计分析，并加入失败模式的定性分析。通过在物理机器人上对多种抓取任务的学习策略进行评估，验证了这些方法的有效性。
原文摘要 · Abstract (English)
The robot learning community has made great strides in recent years, proposing new architectures and showcasing impressive new capabilities; however, the dominant metric used in the literature, especially for physical experiments, is "success rate", i.e. the percentage of runs that were successful. Furthermore, it is common for papers to report this number with little to no information regarding the number of runs, the initial conditions, and the success criteria, little to no narrative description of the behaviors and failures observed, and little to no statistical analysis of the findings. In this paper we argue that to move the field forward, researchers should provide a nuanced evaluation of their methods, especially when evaluating and comparing learned policies on physical robots. To do so, we propose best practices for future evaluations: explicitly reporting the experimental conditions, evaluating several metrics designed to complement success rate, conducting statistical analysis, and adding a qualitative description of failures modes. We illustrate these through an evaluation on physical robots of several learned policies for manipulation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。