arXiv:2604.14268cs.CV2026-04被引 25

用文本或单图生成可导航的高保真3D场景,支持多模态输入与交互探索。

HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

论文配图:HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
图 1 · 摘自论文原文
  • 四阶段流程:全景生成→轨迹规划→世界扩展→场景合成
  • 支持单图/文本生成3DGS场景,性能媲美闭源模型Marble
  • 提供可交互3D渲染平台WorldLens,支持角色与实时探索

我们提出HY-World 2.0,一个支持多模态输入(文本、单视图图像、多视图图像、视频)的三维世界建模框架,能生成可导航的3D高斯溅射(3DGS)场景。基于文本或单视图图像输入,模型通过四阶段方法实现世界生成:a) HY-Pano 2.0全景生成,b) WorldNav轨迹规划,c) WorldStereo 2.0关键帧视图生成,d) WorldMirror 2.0场景融合。其中引入新创新:提升全景保真度,增强3D理解与规划能力,并升级关键帧视图生成模型与通用3D预测模型。同时推出高精度3DGS渲染平台WorldLens,具备引擎无关架构、自动IBL光照、高效碰撞检测及训练渲染协同设计,支持角色交互式探索。大量实验表明,该框架在多个开源基准上达到领先水平,性能接近闭源模型Marble。所有模型权重、代码与技术细节均已公开,以促进可复现性与研究推进。

原文摘要 · Abstract (English)

We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.

3D生成多模态世界模型3DGS

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。