arXiv:2503.10684cs.CVcs.AI2025-03ICCV被引 5

从无标注视频中自动拆分技能片段,提升智能体任务完成能力。

Open-World Skill Discovery from Unsegmented Demonstrations

  • 基于预测误差检测技能边界,无需人工标注。
  • 在Minecraft中使短任务性能提升63.7%,长任务提升20.8%。
  • 可直接用YouTube视频训练指令跟随智能体,适合开放世界研究者。

在开放世界环境中学习技能对构建能组合基础技能完成多样任务的智能体至关重要。在线演示视频通常时长较长且未分割,难以进行标注与分割。不同于依赖序列采样或人工标注的方法,本文提出一种基于自监督学习的算法,将长视频自动分割为语义一致、技能连贯的片段。受人类认知事件分割理论启发,引入无需标注的时序视频分割算法Skill Boundary Detection(SBD),通过预训练的无条件动作预测模型的预测误差变化来识别技能边界。当预测误差显著上升时,即表示当前执行的技能发生切换。在Minecraft这一丰富的开放世界模拟器上评估,SBD生成的片段使条件策略在短期原子技能任务上的平均性能提升63.7%和52.1%,对应层次化智能体在长期任务上提升11.3%和20.8%。该方法可利用大量YouTube视频训练指令跟随智能体。

原文摘要 · Abstract (English)

Learning skills in open-world environments is essential for developing agents capable of handling a variety of tasks by combining basic skills. Online demonstration videos are typically long but unsegmented, making them difficult to segment and label with skill identifiers. Unlike existing methods that rely on sequence sampling or human labeling, we have developed a self-supervised learning-based approach to segment these long videos into a series of semantic-aware and skill-consistent segments. Drawing inspiration from human cognitive event segmentation theory, we introduce Skill Boundary Detection (SBD), an annotation-free temporal video segmentation algorithm. SBD detects skill boundaries in a video by leveraging prediction errors from a pretrained unconditional action-prediction model. This approach is based on the assumption that a significant increase in prediction error indicates a shift in the skill being executed. We evaluated our method in Minecraft, a rich open-world simulator with extensive gameplay videos available online. Our SBD-generated segments improved the average performance of conditioned policies by 63.7% and 52.1% on short-term atomic skill tasks, and their corresponding hierarchical agents by 11.3% and 20.8% on long-horizon tasks. Our method can leverage the diverse YouTube videos to train instruction-following agents. The project page can be found in https://craftjarvis.github.io/SkillDiscovery.

技能发现视频分割自监督学习开放世界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。