测试发现,仅看动作的评估指标无法识别影响模仿学习的关键错误。
Auditing Demonstration Curation Metrics: Action-Only Scorers Fail on the Structural Defects That Degrade Imitation Policies
- 通过可控实验注入已知缺陷,测试七种筛选指标的效果
- 结构性错误导致任务失败,但所有只看动作的指标都未发现
- 需分析状态轨迹的指标才能识别错误,但效果仍有限
模仿学习策略继承训练演示的质量,目前已有多种自动评分与筛选低质量演示的评估指标。这些指标在不同数据集和协议下验证,难以判断哪种真正能识别损害策略的演示。本文构建受控测试环境,系统性注入已知类型的演示缺陷,评估七种指标在两个维度上的表现:区分缺陷与正常演示的能力,以及基于其筛选结果训练行为克隆策略后的任务成功率。研究两类缺陷:细微扰动(相关动作噪声、震颤、截断)可被多变量异常检测识别,去除后能恢复全部下游性能差距;而结构性错误(关键时刻执行错误动作)对所有仅关注动作的指标均不可见,甚至出现反向评分——将劣质演示评为高质量,用于筛选反而使策略性能停留在或低于未经筛选的基线。唯有考察状态轨迹的指标能检测结构性错误,但最优者也仅恢复三分之一的性能差距。高检测准确率并不保证下游性能提升。代码与实现均已开源。
原文摘要 · Abstract (English)
Imitation-learning policies inherit the quality of the demonstrations they are trained on, and a growing set of curation metrics promise to score and filter low-quality demonstrations automatically. These metrics are each validated on different data with different protocols, so it is unclear which of them actually identify the demonstrations that harm a policy. We build a controlled testbed in which demonstration defects are injected with known type, and audit seven curation metrics along two axes: how well each separates defective from clean demonstrations, and whether training a behavior-cloning policy on each metric's curated subset improves task success. We study two defect regimes. Subtle perturbations (correlated action noise, tremor, truncation) are detectable by multivariate outlier scoring and, once removed, recover the full downstream gap. Structural errors, where the demonstration executes a wrong action at a key moment, are invisible to every action-only metric we test, and two of them are inverted: they score defective demonstrations as higher quality and, used for curation, tend to leave the policy at or below the uncurated baseline rather than above it. Only metrics that examine the state trajectory detect structural errors, and even the best of them recovers just a third of the downstream gap. High detection accuracy does not guarantee downstream improvement. We release the testbed and all curation implementations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。