在有干扰物的视频中，用少量真实动作标注能显著提升潜在动作学习效果。
Latent Action Learning Requires Supervision in the Presence of Distractors
- 改进LAPO模型，引入少量真实动作监督以增强潜在动作质量。
- 仅用2.5%真实动作数据，下游性能平均提升4.2倍。
- 适合需要从杂乱视频中学习动作的机器人与视觉强化学习研究者。
最近，由潜行动作策略（LAPO）引领的潜行动作学习在仅观察数据上展现出出色的预训练效率，为利用网络上海量视频数据推动具身智能提供了可能。然而，先前工作主要基于无干扰数据，其中观测变化主要由真实动作解释。现实中视频包含与动作相关的干扰物，可能阻碍潜行动作学习。我们使用干扰控制套件（DCS）实证研究干扰物对潜行动作学习的影响，发现LAPO在此场景下表现不佳。为此，我们提出LAOM，一种简单的LAPO改进方法，使潜行动作质量提升8倍（线性探测评估）。更重要的是，仅在训练中加入2.5%全数据量的真实动作监督，即可使下游性能平均提升4.2倍。结果表明，在存在干扰物时，将监督融入潜动模型（LAM）训练至关重要，挑战了先学习LAM再解码至真实动作的传统流程。
原文摘要 · Abstract (English)
Recently, latent action learning, pioneered by Latent Action Policies (LAPO), have shown remarkable pre-training efficiency on observation-only data, offering potential for leveraging vast amounts of video available on the web for embodied AI. However, prior work has focused on distractor-free data, where changes between observations are primarily explained by ground-truth actions. Unfortunately, real-world videos contain action-correlated distractors that may hinder latent action learning. Using Distracting Control Suite (DCS) we empirically investigate the effect of distractors on latent action learning and demonstrate that LAPO struggle in such scenario. We propose LAOM, a simple LAPO modification that improves the quality of latent actions by 8x, as measured by linear probing. Importantly, we show that providing supervision with ground-truth actions, as few as 2.5% of the full dataset, during latent action learning improves downstream performance by 4.2x on average. Our findings suggest that integrating supervision during Latent Action Models (LAM) training is critical in the presence of distractors, challenging the conventional pipeline of first learning LAM and only then decoding from latent to ground-truth actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。