用生成视频让机器人先‘做梦’再行动，自然语言就能指挥它导航。
DreamToNav: Generalizable Navigation for Robots via Generative Video Planning
- 用语言指令生成逼真视频，再从中提取机器人的运动路径。
- 在室内场景中成功率达76.7%，定位误差小于0.1米，轨迹误差低于0.15米。
- 无需为每种任务重写代码，适合想快速部署智能机器人的研究者。
我们提出DreamToNav，一种新型自主机器人框架，利用生成视频模型实现直观的人机协同控制。用户无需设定固定路点，只需输入自然语言指令（如“小心跟随那个人”），系统便将其转化为可执行动作。流程首先使用Qwen 2.5-VL-7B-Instruct将模糊指令转化为精确的视觉描述，再以这些描述为条件，驱动NVIDIA Cosmos 2.5——一个顶尖的视频基础模型，生成物理上一致的机器人执行任务的视频序列。从该合成视频中，通过视觉姿态估计、机器人检测与轨迹恢复技术提取有效运动路径。通过将视频生成作为规划引擎，DreamToNav使机器人能在实际执行前“视觉化地预演”复杂行为，提供统一框架实现避障与目标导向导航，且无需针对具体任务进行工程设计。我们在轮式移动机器人和四足机器人上评估该方法，应用于室内导航任务。结果显示，成功率达76.7%，最终目标误差通常在0.05–0.10米之间，轨迹跟踪误差低于0.15米。这表明从生成视频预测中提取的轨迹可在不同运动平台上可靠执行。
原文摘要 · Abstract (English)
We present DreamToNav, a novel autonomous robot framework that uses generative video models to enable intuitive, human-in-the-loop control. Instead of relying on rigid waypoint navigation, users provide natural language prompts (e.g. ``Follow the person carefully''), which the system translates into executable motion. Our pipeline first employs Qwen 2.5-VL-7B-Instruct to refine vague user instructions into precise visual descriptions. These descriptions condition NVIDIA Cosmos 2.5, a state-of-the-art video foundation model, to synthesize a physically consistent video sequence of the robot performing the task. From this synthetic video, we extract a valid kinematic path using visual pose estimation, robot detection and trajectory recovery. By treating video generation as a planning engine, DreamToNav allows robots to visually "dream" complex behaviors before executing them, providing a unified framework for obstacle avoidance and goal-directed navigation without task-specific engineering. We evaluate the approach on both a wheeled mobile robot and a quadruped robot in indoor navigation tasks. DreamToNav achieves a success rate of 76.7%, with final goal errors typically within 0.05-0.10 m and trajectory tracking errors below 0.15 m. These results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。