用统一多模态模型实现端到端决策规划,提升智能体任务成功率。
Planning with Unified Multimodal Models
- 基于统一多模态模型构建端到端规划框架,集成策略、动态模型与价值函数。
- 在具身决策任务中,成功率显著高于纯视觉语言模型方法。
- 无需专家示范,数据效率高,适合追求高效决策的AI研究者。
随着大语言模型(LLMs)和视觉语言模型(VLMs)强大推理能力的发展,近期研究探索将其用于决策。然而,多数方法仅依赖语言推理,限制了其决策能力。统一多模态模型(UMMs)支持多模态输入与输出,为通过生成视觉内容进行推理提供了新路径。为此,我们提出Uni-Plan,一个基于UMMs的规划框架。该框架中单一模型同时充当策略、动态模型与价值函数。为避免动态预测中的幻觉问题,我们提出自判别过滤机制,由生成模型自身作为判别器过滤无效预测。在具身决策任务上的实验表明,Uni-Plan显著优于基于VLM的方法,且具备强数据可扩展性:无需专家示范,在相同训练数据量下表现更优。本工作为未来基于UMMs的推理与决策研究奠定基础。
原文摘要 · Abstract (English)
With the powerful reasoning capabilities of large language models (LLMs) and vision-language models (VLMs), many recent works have explored using them for decision-making. However, most of these approaches rely solely on language-based reasoning, which limits their ability to reason and make informed decisions. Recently, a promising new direction has emerged with unified multimodal models (UMMs), which support both multimodal inputs and outputs. We believe such models have greater potential for decision-making by enabling reasoning through generated visual content. To this end, we propose Uni-Plan, a planning framework built on UMMs. Within this framework, a single model simultaneously serves as the policy, dynamics model, and value function. In addition, to avoid hallucinations in dynamics predictions, we present a novel approach self-discriminated filtering, where the generative model serves as a self-discriminator to filter out invalid dynamics predictions. Experiments on embodied decision-making tasks show that Uni-Plan substantially improves success rates compared to VLM-based methods, while also showing strong data scalability, requiring no expert demonstrations and achieving better performance under the same training-data size. This work lays a foundation for future research in reasoning and decision-making with UMMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。