用视觉语言模型增强机器人抓取规划,让机械臂更懂环境和指令。
Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation
- 结合视觉语言模型生成动作序列,再用预测模型模拟未来状态。
- 在真实场景中完成复杂抓取任务,性能优于现有方法。
- 适合需要理解指令与环境的智能机器人系统开发。
模型预测控制(MPC)广泛用于机器人规划与控制,但缺乏环境感知能力,在复杂非结构化场景中易失效。为此,本文提出视觉语言模型预测控制(VLMPC),将视觉语言模型(VLM)的感知能力与MPC结合。VLMPC通过条件动作采样模块,以目标图像或语言指令为输入,利用VLM生成候选动作序列,并输入视频预测模型以模拟未来帧。为进一步降低计算开销,提出改进版Traj-VLMPC,以运动轨迹生成替代视频预测,保持精度的同时提升效率。两者均采用基于VLM的分层代价函数,综合考虑像素级与知识级一致性,选择最优动作序列。实验表明,两种方法在公开基准测试中均超越现有先进方法,并在多种真实机器人抓取任务中表现优异。代码已开源。
原文摘要 · Abstract (English)
Model Predictive Control (MPC) is a widely adopted control paradigm that leverages predictive models to estimate future system states and optimize control inputs accordingly. However, while MPC excels in planning and control, it lacks the capability for environmental perception, leading to failures in complex and unstructured scenarios. To address this limitation, we introduce Vision-Language Model Predictive Control (VLMPC), a robotic manipulation planning framework that integrates the perception power of vision-language models (VLMs) with MPC. VLMPC utilizes a conditional action sampling module that takes a goal image or language instruction as input and leverages VLM to generate candidate action sequences. These candidates are fed into a video prediction model that simulates future frames based on the actions. In addition, we propose an enhanced variant, Traj-VLMPC, which replaces video prediction with motion trajectory generation to reduce computational complexity while maintaining accuracy. Traj-VLMPC estimates motion dynamics conditioned on the candidate actions, offering a more efficient alternative for long-horizon tasks and real-time applications. Both VLMPC and Traj-VLMPC select the optimal action sequence using a VLM-based hierarchical cost function that captures both pixel-level and knowledge-level consistency between the current observation and the task input. We demonstrate that both approaches outperform existing state-of-the-art methods on public benchmarks and achieve excellent performance in various real-world robotic manipulation tasks. Code is available at https://github.com/PPjmchen/VLMPC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。