构建新数据集与框架，提升大模型在真实环境中的规划能力
Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models
- 用多模态数据训练模型理解环境状态并生成行动计划
- 发现GPT-4V等模型在感知与推理上存在明显瓶颈
- 结合神经网络与符号推理，提升复杂任务规划准确性
大型多模态模型在视觉与语言任务中展现出强大的问题解决能力，并具备编码广泛世界知识的潜力。然而，这些模型在现实环境中进行感知、推理、规划和执行仍面临挑战。本文提出Can-Do，一个用于评估具身规划能力的新基准数据集，包含400个多模态样本，每条样本包括自然语言指令、环境图像、状态变化及对应动作计划，涵盖常识、物理理解与安全意识等多个方面。细粒度分析显示，当前先进模型如GPT-4V在视觉感知、理解与推理方面存在明显局限。为此，我们提出NeuroGround——一种神经符号框架，先将计划生成锚定于感知到的环境状态，再利用符号规划引擎增强模型生成的计划。实验表明该框架显著优于强基线。代码与数据集已公开于https://embodied-planning.github.io。
原文摘要 · Abstract (English)
Large multimodal models have demonstrated impressive problem-solving abilities in vision and language tasks, and have the potential to encode extensive world knowledge. However, it remains an open challenge for these models to perceive, reason, plan, and act in realistic environments. In this work, we introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities through more diverse and complex scenarios than previous datasets. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. The data encompasses diverse aspects of commonsense knowledge, physical understanding, and safety awareness. Our fine-grained analysis reveals that state-of-the-art models, including GPT-4V, face bottlenecks in visual perception, comprehension, and reasoning abilities. To address these challenges, we propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans. Experimental results demonstrate the effectiveness of our framework compared to strong baselines. Our code and dataset are available at https://embodied-planning.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。