多智能体框架让机器人能闭环执行语言指令，自适应环境变化。
MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation
- 分角色协作：分解、定位、思考、反思四智能体协同决策。
- 闭环反馈提升成功率：真实场景零样本任务成功率显著提高。
- 错误可修复：仅重激活相关智能体，避免重复规划开销。
基于大语言模型（LLM）的任务规划在机器人操作中兴起，但以往方法依赖专用模型微调或提示工程，常采用开环模式，缺乏环境反馈，在动态环境中脆弱。MALLVI提出多智能体大语言与视觉框架，实现闭环反馈驱动的机器人操作。给定自然语言指令和环境图像，系统生成可执行的原子动作；执行后，视觉语言模型（VLM）评估环境反馈，决定是否重复或推进。不依赖单模型，而是协调分解器、定位器、思考者和反思者四类专用智能体，分别处理感知、定位、推理与高层规划。可选描述器智能体保存初始状态视觉记忆。反思者通过精准触发相关智能体实现错误检测与恢复，避免全量重规划。仿真与真实世界实验表明，迭代式闭环多智能体协同提升了泛化能力，并显著提高零样本操作任务的成功率。代码已开源。
原文摘要 · Abstract (English)
Task planning for robotic manipulation with large language models (LLMs) is an emerging area. Prior approaches rely on specialized models, fine tuning, or prompt tuning, and often operate in an open loop manner without robust environmental feedback, making them fragile in dynamic settings. MALLVI presents a Multi Agent Large Language and Vision framework that enables closed-loop feedback driven robotic manipulation. Given a natural language instruction and an image of the environment, MALLVI generates executable atomic actions for a robot manipulator. After action execution, a Vision Language Model (VLM) evaluates environmental feedback and decides whether to repeat the process or proceed to the next step. Rather than using a single model, MALLVI coordinates specialized agents, Decomposer, Localizer, Thinker, and Reflector, to manage perception, localization, reasoning, and high level planning. An optional Descriptor agent provides visual memory of the initial state. The Reflector supports targeted error detection and recovery by reactivating only relevant agents, avoiding full replanning. Experiments in simulation and real-world settings show that iterative closed loop multi agent coordination improves generalization and increases success rates in zero shot manipulation tasks. Code available at https://github.com/iman1234ahmadi/MALLVI .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。