让AI理解视觉与语言，还能预判错误情况，做出更可靠的计划。
LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning
- 用视觉语言模型结合图文生成可执行动作序列
- 在两个基准上任务完成率和逻辑连贯性均超越现有模型
- 适合需要推理错误场景的智能机器人系统研究
尽管大型语言模型（LLMs）凭借强大的推理能力推动了具身AI系统的程序规划发展，但多模态输入与反事实推理的融合仍鲜有探索。为此，我们提出LLaPa，一种面向多模态程序规划的视觉-语言模型框架。该框架通过视觉语言模型（VLMs），从文本任务描述和视觉环境图像中生成可执行的动作序列。此外，我们引入两个辅助模块提升规划性能：第一个是任务-环境重排序器（TER），利用任务导向分割构建任务敏感特征空间，使文本描述与视觉环境对齐并突出执行关键区域；第二个是反事实行为检索器（CAR），识别并强调潜在的反事实条件，增强模型在反事实场景中的推理能力。在ActPlan-1K和ALFRED基准上的大量实验表明，LLaPa生成的计划质量更高，逻辑连贯性（LCS）和正确性均优于先进模型。代码与模型已公开于https://github.com/sunshibo1234/LLaPa。
原文摘要 · Abstract (English)
While large language models (LLMs) have advanced procedural planning for embodied AI systems through strong reasoning abilities, the integration of multimodal inputs and counterfactual reasoning remains underexplored. To tackle these challenges, we introduce LLaPa, a vision-language model framework designed for multimodal procedural planning. LLaPa generates executable action sequences from textual task descriptions and visual environmental images using vision-language models (VLMs). Furthermore, we enhance LLaPa with two auxiliary modules to improve procedural planning. The first module, the Task-Environment Reranker (TER), leverages task-oriented segmentation to create a task-sensitive feature space, aligning textual descriptions with visual environments and emphasizing critical regions for procedural execution. The second module, the Counterfactual Activities Retriever (CAR), identifies and emphasizes potential counterfactual conditions, enhancing the model's reasoning capability in counterfactual scenarios. Extensive experiments on ActPlan-1K and ALFRED benchmarks demonstrate that LLaPa generates higher-quality plans with superior LCS and correctness, outperforming advanced models. The code and models are available https://github.com/sunshibo1234/LLaPa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。