仅用视频训练模型，就能学会复杂规则与规划能力。
VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
- 用无标签视频训练自回归生成模型，从视觉变化中学习知识。
- 3亿参数模型在视频围棋任务达5段水平，无需强化学习奖励或搜索。
- 适用于机器人控制等需要视觉推理的场景，可开源复现。
本研究探索深度生成模型能否仅通过视觉输入学习复杂知识，而非依赖文本模型。我们提出 VideoWorld，一个基于无标签视频数据训练的自回归视频生成模型，并在视频围棋和机器人控制任务中测试其知识获取能力。实验发现：（1）仅视频训练已足够学习规则、推理与规划能力；（2）视觉变化表征对知识获取至关重要。为此，我们引入潜变量动态模型（LDM）作为核心组件。令人惊讶的是，仅用3亿参数的 VideoWorld 在 Video-GoBench 上达到5段职业水平，且无需强化学习中的搜索算法或奖励机制。在机器人任务中，VideoWorld 能有效学习多种控制操作，并在 CALVIN 与 RLBench 上逼近理想模型表现。该研究为从视觉数据中获取知识开辟新路径，所有代码、数据与模型均已开源。
原文摘要 · Abstract (English)
This work explores whether a deep generative model can learn complex knowledge solely from visual input, in contrast to the prevalent focus on text-based models like large language models (LLMs). We develop VideoWorld, an auto-regressive video generation model trained on unlabeled video data, and test its knowledge acquisition abilities in video-based Go and robotic control tasks. Our experiments reveal two key findings: (1) video-only training provides sufficient information for learning knowledge, including rules, reasoning and planning capabilities, and (2) the representation of visual change is crucial for knowledge acquisition. To improve both the efficiency and efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional level in the Video-GoBench with just a 300-million-parameter model, without relying on search algorithms or reward mechanisms typical in reinforcement learning. In robotic tasks, VideoWorld effectively learns diverse control operations and generalizes across environments, approaching the performance of oracle models in CALVIN and RLBench. This study opens new avenues for knowledge acquisition from visual data, with all code, data, and models open-sourced for further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。