用代码规划+语义图状态,让机器人长程操作更可靠
CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models

- 构建持久语义图记录关键实体关系,应对部分可观测场景
- 代码规划器生成子任务指令,减少视觉干扰并提升执行效率
- 适合复杂现实场景中需长期推理的机器人操作任务
视觉-语言-动作(VLA)模型有望实现通用机器人操作,但通常作为短时序策略训练与部署,假设最新观测足以支持动作推理。这一假设在非马尔可夫的长时序任务中失效,因任务相关证据可能被遮挡或早期出现,且杂乱环境使细粒度视觉定位变得脆弱。本文提出CodeGraphVLP,一种分层框架,通过结合持久的语义图状态、可执行代码规划器及进度引导的视觉-语言提示,实现可靠的长时序操作。语义图在部分可观测条件下持续维护任务相关实体及其关系;合成规划器基于该图进行高效进展检查,并输出子任务指令及关联物体。利用这些输出构建抑制杂乱的观测,聚焦于关键证据。在真实世界非马尔可夫任务上,CodeGraphVLP优于强基线和历史增强变体,同时显著降低规划延迟,相比端到端视觉语言模型规划。我们还进行了大量消融实验,验证各组件贡献。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models promise generalist robot manipulation, but are typically trained and deployed as short-horizon policies that assume the latest observation is sufficient for action reasoning. This assumption breaks in non-Markovian long-horizon tasks, where task-relevant evidence can be occluded or appear only earlier in the trajectory, and where clutter and distractors make fine-grained visual grounding brittle. We present CodeGraphVLP, a hierarchical framework that enables reliable long-horizon manipulation by combining a persistent semantic-graph state with an executable code-based planner and progress-guided visual-language prompting. The semantic-graph maintains task-relevant entities and relations under partial observability. The synthesized planner executes over this semantic-graph to perform efficient progress checks and outputs a subtask instruction together with subtask-relevant objects. We use these outputs to construct clutter-suppressed observations that focus the VLA executor on critical evidence. On real-world non-Markovian tasks, CodeGraphVLP improves task completion over strong VLA baselines and history-enabled variants while substantially lowering planning latency compared to VLM-in-the-loop planning. We also conduct extensive ablation studies to confirm the contributions of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。