统一3D场景理解与未来几何预测,提升自动驾驶模拟能力。
HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

- 用BEV表示融合多视角信息,适配大模型推理
- 通过语义上下文预测未来点云,准确率超越专用模型
- 适合自动驾驶系统研发与多模态建模研究者
驾驶世界模型是自动驾驶的核心技术,用于模拟环境动态。现有方法多聚焦未来场景生成,忽视全面的3D场景理解;而大语言模型虽具强大推理能力,却无法预测几何演化,导致语义理解与物理模拟脱节。为此,我们提出HERMES++,一个集成3D场景理解与未来几何预测的统一框架。首先,采用鸟瞰图(BEV)表示将多视角空间信息整合为适配大模型的结构;其次,引入增强型世界查询,实现理解分支的知识迁移;第三,设计当前到未来的关联机制,使几何演化依赖语义上下文;最后,通过联合几何优化策略,结合显式几何约束与隐式潜在正则化,确保内部表征符合几何先验。在多个基准测试中验证表明,HERMES++在点云未来预测与3D场景理解任务上均优于专用方法。代码与模型将公开于https://github.com/H-EmbodVis/HERMESV2。
原文摘要 · Abstract (English)
Driving world models serve as a pivotal technology for autonomous driving by simulating environmental dynamics. However, existing approaches predominantly focus on future scene generation, often overlooking comprehensive 3D scene understanding. Conversely, while Large Language Models (LLMs) demonstrate impressive reasoning capabilities, they lack the capacity to predict future geometric evolution, creating a significant disparity between semantic interpretation and physical simulation. To bridge this gap, we propose HERMES++, a unified driving world model that integrates 3D scene understanding and future geometry prediction within a single framework. Our approach addresses the distinct requirements of these tasks through synergistic designs. First, a BEV representation consolidates multi-view spatial information into a structure compatible with LLMs. Second, we introduce LLM-enhanced world queries to facilitate knowledge transfer from the understanding branch. Third, a Current-to-Future Link is designed to bridge the temporal gap, conditioning geometric evolution on semantic context. Finally, to enforce structural integrity, we employ a Joint Geometric Optimization strategy that integrates explicit geometric constraints with implicit latent regularization to align internal representations with geometry-aware priors. Extensive evaluations on multiple benchmarks validate the effectiveness of our method. HERMES++ achieves strong performance, outperforming specialist approaches in both future point cloud prediction and 3D scene understanding tasks. The model and code will be publicly released at https://github.com/H-EmbodVis/HERMESV2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。