用分割模型聚焦主体,让动作模型忽略干扰,提升真实视频预训练效果。
Segment to Focus: Guiding Latent Action Models in the Presence of Distractors
- 通过掩码只重建主体像素,强制潜行动作关注可控动态。
- 在两个基准上,误差降低3.5倍,回报提升4.97倍,接近有标签方法。
- 无需额外标注或模型修改,可零样本适配现成分割模型。
潜行动作模型(LAMs)为在无动作标签视频上预训练具身智能体提供了新路径。它们从连续观测中推断潜行动作,后续仅需少量标签即可解码为真实动作。然而,当视频中存在与动作相关的视觉干扰(如动态背景、相机抖动或移动物体)时,标准重建目标会引导潜行动作编码外部运动而非代理控制的动态,导致微调后策略性能下降。我们观察到,内生因素与外生因素在像素空间中通常具有空间分离性:控制相关变化集中在代理本身,而干扰运动发生在其他区域。为此,我们提出MaskLAM方法,仅对代理像素施加重建约束,迫使潜行动作用于解释代理控制的动力学而非外部干扰。该方法零样本使用现成分割模型(如SAM)获取代理掩码,无需架构改动、辅助损失或动作标签。在两个连续控制基准(Distracting Control Suite, Distracting Meta-World)上,MaskLAM相比LAPO将归一化线性探测均方误差降低最多3.51倍,归一化回报提升最多4.97倍,显著缩小与依赖真值动作监督的LAOM-Labels之间的差距。
原文摘要 · Abstract (English)
Latent action models (LAMs) offer a promising path to pre-training embodied agents on large amounts of action-free video. They infer latent actions between consecutive observations that can later be decoded to ground-truth actions using a small number of labels. However, recent work has shown that this recipe fails in the presence of action-correlated visual distractors common in real-world video, such as dynamic backgrounds, camera shake, or other moving objects. In these scenarios, the standard reconstruction objective drives latent actions to encode exogenous motion instead of agent-controlled dynamics, resulting in policies that underperform when fine-tuned. We observe, however, that endogenous and exogenous factors are typically spatially separated in pixel space: control-relevant change is concentrated on the agent, while distractor motion occurs elsewhere. We exploit this observation by restricting the reconstruction objective to agent pixels, forcing latent actions to explain agent-controlled dynamics rather than exogenous ones. We call this method MaskLAM; it obtains the agent mask zero-shot from off-the-shelf segmentation foundation models (e.g., SAM) and requires no architectural changes, auxiliary losses, or action labels during pre-training. Across two continuous-control benchmarks (Distracting Control Suite, Distracting Meta-World), MaskLAM reduces normalized linear-probe MSE by up to $3.51\times$ and improves normalized return by up to $4.97\times$ over LAPO, while narrowing the gap to LAOM-Labels, which relies on ground-truth action supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。