首次系统整合2D到4D多模态生成,构建真实世界模拟统一框架
Simulating the Real World: A Unified Survey of Multimodal Generative Models

- 按数据维度演进:从2D图像→视频→3D→4D,逐层融合外观、动态与几何
- 梳理跨模态数据集与评估指标,填补多维生成研究的整合空白
- 适合对世界模型、生成式AI和具身智能感兴趣的科研人员
实现对真实世界的理解与复现是通用人工智能(AGI)研究的核心挑战。现有方法如世界模型试图捕捉物理世界的基本规律,以实现更精准的模拟与交互。然而,当前方法常将2D图像、视频、3D及4D表示视为独立领域,忽视其内在关联;且多聚焦于单一维度,缺乏系统性整合。本文首次提出一个统一的多模态生成模型综述框架,涵盖从2D生成(外观)到视频(外观+动态)、3D生成(外观+几何),最终至4D生成(全维度融合)的演进路径。该综述系统梳理了相关数据集、评估指标与未来方向,为研究者提供清晰指引,助力多模态生成与真实世界模拟的协同发展。
原文摘要 · Abstract (English)
Understanding and replicating the real world is a critical challenge in Artificial General Intelligence (AGI) research. To achieve this, many existing approaches, such as world models, aim to capture the fundamental principles governing the physical world, enabling more accurate simulations and meaningful interactions. However, current methods often treat different modalities, including 2D (images), videos, 3D, and 4D representations, as independent domains, overlooking their interdependencies. Additionally, these methods typically focus on isolated dimensions of reality without systematically integrating their connections. In this survey, we present a unified survey for multimodal generative models that investigate the progression of data dimensionality in real-world simulation. Specifically, this survey starts from 2D generation (appearance), then moves to video (appearance+dynamics) and 3D generation (appearance+geometry), and finally culminates in 4D generation that integrate all dimensions. To the best of our knowledge, this is the first attempt to systematically unify the study of 2D, video, 3D and 4D generation within a single framework. To guide future research, we provide a comprehensive review of datasets, evaluation metrics and future directions, and fostering insights for newcomers. This survey serves as a bridge to advance the study of multimodal generative models and real-world simulation within a unified framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。