让机器人自检自修代码,提升任务完成率。
HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
- 融合代码生成与视觉语言反馈,构建闭环编程循环。
- 在模拟环境中实现失败检测与自动修复,成功率显著提升。
- 适合需要自主决策的智能机器人系统开发人员。
多模态大语言模型(MLLM)的进步为具身智能体的代码策略生成提供了更丰富的感知基础。然而,现有系统大多缺乏有效机制来适应性地监控策略执行并修复代码。本文提出HyCodePolicy,一种基于混合语言控制的框架,将代码合成、几何定位、感知监控和迭代修复整合进具身智能体的闭环编程周期。给定自然语言指令后,系统首先将其分解为子目标,并生成基于物体中心几何基元的可执行程序。程序在模拟环境中执行,同时视觉-语言模型(VLM)观察关键节点,检测并定位执行失败,推断失败原因。通过融合结构化执行轨迹与VLM感知反馈,HyCodePolicy推断失败原因并修复代码。该混合双反馈机制实现了无需人工干预的自我纠错式程序生成。实验表明,HyCodePolicy显著提升了机器人操作策略的鲁棒性和样本效率,为多模态推理集成到自主决策流水线提供了可扩展方案。
原文摘要 · Abstract (English)
Recent advances in multimodal large language models (MLLMs) have enabled richer perceptual grounding for code policy generation in embodied agents. However, most existing systems lack effective mechanisms to adaptively monitor policy execution and repair codes during task completion. In this work, we introduce HyCodePolicy, a hybrid language-based control framework that systematically integrates code synthesis, geometric grounding, perceptual monitoring, and iterative repair into a closed-loop programming cycle for embodied agents. Technically, given a natural language instruction, our system first decomposes it into subgoals and generates an initial executable program grounded in object-centric geometric primitives. The program is then executed in simulation, while a vision-language model (VLM) observes selected checkpoints to detect and localize execution failures and infer failure reasons. By fusing structured execution traces capturing program-level events with VLM-based perceptual feedback, HyCodePolicy infers failure causes and repairs programs. This hybrid dual feedback mechanism enables self-correcting program synthesis with minimal human supervision. Our results demonstrate that HyCodePolicy significantly improves the robustness and sample efficiency of robot manipulation policies, offering a scalable strategy for integrating multimodal reasoning into autonomous decision-making pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。