融合大模型与世界模型,推动智能体在真实世界中自主决策。
Embodied AI: From LLMs to World Models
- 结合大语言模型与世界模型,实现从语言指令到物理动作的端到端推理。
- 通过内部表征与未来预测,使智能体在真实环境中符合物理规律交互。
- 适合对具身智能、多模态感知与通用人工智能感兴趣的科研人员。
具身人工智能(Embodied AI)是实现通用人工智能(AGI)的重要范式,正推动虚拟空间向物理系统演进。近期大型语言模型(LLMs)与世界模型(WMs)的突破为具身智能带来新机遇:一方面,LLMs通过语义推理和任务分解,将自然语言指令与动作融入认知;另一方面,WMs构建对外部世界的内部表征与未来预测,支持符合物理规律的具身交互。本文系统梳理了具身智能的发展脉络,涵盖基础技术、核心组件与硬件系统,并从单模态到多模态视角分析其演进。重点探讨两类前沿方向:基于大模型/多模态大模型(MLLMs)的具身智能与基于世界模型的具身智能,阐明二者在端到端认知与物理驱动交互中的关键作用。在此基础上,提出联合MLLM-WM架构的必要性,揭示其在复杂物理任务中的深远意义。此外,展示代表性应用案例,验证其在现实场景中的广泛适用性,并指出未来值得深入研究的方向。
原文摘要 · Abstract (English)
Embodied Artificial Intelligence (AI) is an intelligent system paradigm for achieving Artificial General Intelligence (AGI), serving as the cornerstone for various applications and driving the evolution from cyberspace to physical systems. Recent breakthroughs in Large Language Models (LLMs) and World Models (WMs) have drawn significant attention for embodied AI. On the one hand, LLMs empower embodied AI via semantic reasoning and task decomposition, bringing high-level natural language instructions and low-level natural language actions into embodied cognition. On the other hand, WMs empower embodied AI by building internal representations and future predictions of the external world, facilitating physical law-compliant embodied interactions. As such, this paper comprehensively explores the literature in embodied AI from basics to advances, covering both LLM driven and WM driven works. In particular, we first present the history, key technologies, key components, and hardware systems of embodied AI, as well as discuss its development via looking from unimodal to multimodal angle. We then scrutinize the two burgeoning fields of embodied AI, i.e., embodied AI with LLMs/multimodal LLMs (MLLMs) and embodied AI with WMs, meticulously delineating their indispensable roles in end-to-end embodied cognition and physical laws-driven embodied interactions. Building upon the above advances, we further share our insights on the necessity of the joint MLLM-WM driven embodied AI architecture, shedding light on its profound significance in enabling complex tasks within physical worlds. In addition, we examine representative applications of embodied AI, demonstrating its wide applicability in real-world scenarios. Last but not least, we point out future research directions of embodied AI that deserve further investigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。