用时间扭曲自监督学习动作进度，提升机器人模仿学习的数据效率
WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation
- 通过时间扭曲生成帧间进度目标，实现无标注的自监督训练
- 在折叠T恤任务中，数据质量下降时仍保持19/20成功率，较传统方法提升18倍吞吐量
- 适用于含低质量演示的长程任务，适合工业级机器人数据清洗场景
扩展模仿学习依赖大规模数据集，但人类远程操控不可避免产生包含犹豫和恢复的混合质量示范。已有基于帧级进度的奖励模型依赖绝对进度标签，易受噪声干扰，或需高成本人工标注子任务边界。本文提出WARP（Warp-Augmented Relative Progress），一种完全自监督算法，可直接从成功示范中学习密集、带符号的相对进度信号。WARP通过时间扭曲（变速与反转）生成示范的增强版本，并训练WARP-RM预测输入帧间的归一化耗时。将这些预测在重叠窗口上聚合，得到稠密帧级进度信号。进一步提出WARP-BC，利用这些标量奖励估计，在行为克隆中对高优势动作片段进行加权，其中片段优势由帧级奖励聚合获得。我们在物理双臂机器人系统上评估该方法，执行长程可变形物体操作任务：从随机褶皱状态折叠T恤。为评估策略对低质数据的鲁棒性，我们以轨迹长度作为远程操控次优性的代理，构建不同质量的训练数据集。随着数据集放宽容忍更多低效行为，WARP-BC维持19/20成功率，而传统行为克隆降至2/20，吞吐量最高提升18倍。此外，我们在真实世界及可复现仿真环境中评估瓶入箱放置任务，成功度、速度与吞吐量均通过配对显著性检验，所有仿真代码与评估数据已公开。
原文摘要 · Abstract (English)
Scaling imitation learning requires large datasets, yet human teleoperation inevitably produces mixed-quality demonstrations containing hesitations and recoveries. Prior frame-level progress reward models supervise on absolute temporal progress proxies that suffer from label noise, or require costly human annotations to define subtask boundaries. We present WARP (Warp-Augmented Relative Progress), a novel fully self-supervised algorithm for learning dense, signed relative progress magnitudes directly from successful demonstrations. WARP generates per-frame progress targets via time-warp augmentations of demonstrations (variable playback speeds and reversals) and we train WARP-RM to predict the normalized elapsed time between input frames. Aggregating these predictions across overlapping windows yields a dense frame-level progress signal. We then introduce WARP-BC, which leverages these scalar reward estimates to upweight high-advantage action chunks during behavior cloning, where chunk-level advantage is obtained by aggregating per-frame rewards. We evaluate our approach on a physical bimanual robot system performing a long-horizon deformable object manipulation task: folding T-shirts from a random crumpled start. To evaluate policy robustness against suboptimal data, we construct training datasets of varying quality using episode length as a proxy for teleoperation sub-optimality. As the dataset is widened to admit more inefficiencies, WARP-BC maintains a 19/20 success rate compared to vanilla BC's collapse to 2/20, improving throughput by up to 18x. Furthermore, we evaluate a bottle-in-bin placement task in the real-world, as well as in a reproducible simulation of the task, where gains in success, speed, and throughput hold under paired significance tests, and we release all simulation code and evaluation artifacts. Project page: https://uynitsuj.github.io/warp-rm/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。