用视觉语言模型自动生成注意力图,提升视觉模仿学习的数据效率。
AutoFocus-IL: VLM-based Saliency Maps for Data-Efficient Visual Imitation Learning without Extra Human Annotations
- 利用VLM自动识别演示中的关键物体,生成时序注意力图。
- 在CARLA和真实机器人任务中,性能超越标准行为克隆和依赖人工标注的基线。
- 无需额外人工标注,适合资源受限的视觉模仿学习场景。
AutoFocus-IL是一种简单有效的视觉模仿学习方法,通过引导策略关注任务相关特征而非干扰项和虚假关联,提升数据效率与泛化能力。尽管显著性正则化已被证明是有效手段,但现有方法通常依赖昂贵的人类注视数据或手动标注。相比之下,AutoFocus-IL利用视觉语言模型(VLM)自动识别并跟踪示范中的关键物体,生成时序显著性图,突出因果视觉信号并抑制干扰项。这些图用于正则化行为克隆策略,增强视觉注意力与任务相关线索的一致性。在CARLA模拟器和真实机器人操作任务中的实验表明,AutoFocus-IL不仅优于标准行为克隆,还超越了依赖人类监督(如注视数据)的先进基线。代码、数据集及训练策略视频见https://AutoFocus-IL.github.io/。
原文摘要 · Abstract (English)
AutoFocus-IL is a simple yet effective method to improve data efficiency and generalization in visual imitation learning by guiding policies to attend to task-relevant features rather than distractors and spurious correlations. Although saliency regularization has emerged as a promising way to achieve this, existing approaches typically require costly supervision such as human gaze data or manual saliency annotations. In contrast, AutoFocus-IL leverages vision-language models (VLMs) to automatically identify and track key objects in demonstrations, generating temporal saliency maps that highlight causal visual signals while suppressing distractors. These maps are then used to regularize behavior cloning policies, yielding stronger alignment between visual attention and task-relevant cues. Experiments in both the CARLA simulator and real-robot manipulation tasks demonstrate that AutoFocus-IL not only outperforms standard behavior cloning but also surpasses state-of-the-art baselines that assume privileged access to human supervision, such as gaze data. Code, datasets, and trained policy videos are available at https://AutoFocus-IL.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。