arXiv:2512.00453cs.ROcs.AI2025-12

用统计方法减少专家标注需求,让机器人学得更快更省

Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Prediction

  • 根据状态新颖性决定是否请求专家,仅在罕见状态下查询
  • 相比传统方法减少96%专家查询,且在多种任务中表现接近专家水平
  • 无需实时专家介入,适合复杂仿真和机器人部署

主动模仿学习(AIL)通过训练中查询专家来缓解协变量偏移问题。然而,在GPU密集型模拟器、人机协同场景以及重复访问近似状态的机器人集群中,专家动作标注常成为主要成本。本文提出基于置信预测的主动模仿学习查询策略(CRSAIL),仅在当前状态在专家数据集中代表性不足时请求专家标注。CRSAIL通过计算状态与第K个最近专家状态的距离来衡量新颖性,并利用置信预测设定单一全局阈值——该阈值为在线校准得分的经验(1-α)分位数,实现无需分布假设的校准,使α可作为与任务无关的调参参数。此状态空间查询策略对异常值鲁棒,且无需实时专家接管:可先用学习器完整运行轨迹,再对部分状态事后查询专家。在MuJoCo机器人任务上的评估显示,CRSAIL在保持或超越专家级奖励的同时,相较DAgger减少最多96%的专家查询,相较先前AIL方法减少最多65%,且对α和K具有实证鲁棒性,便于在动态未知的新系统上部署。

原文摘要 · Abstract (English)

Active imitation learning (AIL) combats covariate shift by querying an expert during training. However, expert action labeling often dominates the cost, especially in GPU-intensive simulators, human-in-the-loop settings, and robot fleets that revisit near-duplicate states. We present Conformalized Rejection Sampling for Active Imitation Learning (CRSAIL), a querying rule that requests an expert action only when the visited state is under-represented in the expert-labeled dataset. CRSAIL scores state novelty by the distance to the $K$-th nearest expert state and sets a single global threshold via conformal prediction. This threshold is the empirical $(1-α)$ quantile of on-policy calibration scores, providing a distribution-free calibration rule that links $α$ to the expected query rate and makes $α$ a task-agnostic tuning knob. This state-space querying strategy is robust to outliers and, unlike safety-gate-based AIL, can be run without real-time expert takeovers: we roll out full trajectories (episodes) with the learner and only afterward query the expert on a subset of visited states. Evaluated on MuJoCo robotics tasks, CRSAIL matches or exceeds expert-level reward while reducing total expert queries by up to 96% vs. DAgger and up to 65% vs. prior AIL methods, with empirical robustness to $α$ and $K$, easing deployment on novel systems with unknown dynamics.

主动模仿专家查询强化学习高效学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。