arXiv:2605.25077cs.CV2026-05被引 2

让视频世界模型从看场景转向操作物体,支持点击+描线控制物体移动。

WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models

论文配图:WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models
图 1 · 摘自论文原文
  • 用世界坐标系重映射用户画的轨迹,分离物体运动与镜头变化
  • 通过空间路径LoRA注入控制信号,不破坏原有镜头导航能力
  • 轨迹锚定状态持久化,物体离屏后仍能准确回归位置

近期基于视频的世界模型已实现像素空间下以相机导航为主的交互:用户可移动视角,模型生成连贯视觉延续。但其动作空间仍不完整——仅能移动镜头,无法对单个物体进行操作。由于真实交互本质是物体中心的,这类模型更像被动观察者而非可操控环境。本文提出WorldCraft框架,将交互能力从相机导航扩展至物体级轨迹操作。给定用户点击和手绘路径,WorldCraft生成物体沿指定轨迹运动的未来帧,同时相机持续导航场景。核心机制包括:1)归一化世界轨迹(NWT)在相机无关的世界坐标系中表示用户绘制的运动,并根据当前相机姿态动态重投影,分离物体运动与屏幕位移;2)空间路径LoRA(SP-LoRA)通过模型的空间控制路径注入世界空间信号,实现物体操作的同时保留预训练相机控制器;3)轨迹锚定状态持久化(TASP)将世界轨迹视为持久空间状态,在轨迹条件生成后刷新自回归记忆,使移动物体在离开视野后仍能在后续帧中准确重现。实验表明,WorldCraft实现了精准物体控制,在纯相机评估中保持视频世界模型的相机保真度,并在长序列生成中维持物体状态跨镜头迁移能力。

原文摘要 · Abstract (English)

Recent video-based world models have made pixel-space environments interactive at the camera level: users can navigate viewpoints while the model generates coherent visual continuations. Yet their action spaces remain incomplete: users can move the camera, but cannot act on individual objects. Since real-world interaction is inherently object-centric, such models remain closer to passive scene observers than truly manipulable environments. We present WorldCraft, a framework that expands interactive video world models from camera navigation to object-level trajectory actions. Given a user click and a sketched path, WorldCraft generates future frames in which the selected object follows the prescribed trajectory while the camera continues to navigate the scene. WorldCraft achieves this through a trajectory-centric control pipeline: First, Normalized World Trajectory (NWT) represents user-drawn motion in a camera-invariant world coordinate system and dynamically re-projects it under the current camera pose, separating object motion from camera-induced screen-space displacement; Spatial-Pathway LoRA (SP-LoRA) then injects this world-space signal through the model's spatial-control pathway, adding object manipulation capability while preserving the pretrained camera controller; finally, Trajectory-Anchored State Persistence (TASP) treats the world trajectory as a persistent spatial state and refreshes autoregressive memory after trajectory-conditioned generation, allowing moved objects to reappear at their updated positions after leaving the camera view. Experiments show that WorldCraft enables accurate object control, preserves the video-based world model's camera fidelity under camera-only evaluation, and maintains object state across long autoregressive rollouts with off-camera excursions.

视频生成物体控制世界模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。