用3D高斯点云融合4D雷达与单目图像,实现更精准的三维目标检测。
RaGS: Unleashing 3D Gaussian Splatting from 4D Radar and Monocular Cues for 3D Object Detection
- 以3D高斯点云为动态表示,灵活建模场景结构
- 利用4D雷达速度信息提升点云空间分布精度
- 适合自动驾驶中复杂场景下的实时三维感知
4D毫米波雷达是自动驾驶有前景的感知模态,但如何有效融合4D雷达与单目图像进行3D目标检测仍具挑战。现有融合方法或依赖缺乏全局上下文的实例提案,或受限于刚性结构的密集BEV网格,难以适应多样场景。为此,我们提出RaGS,首个利用3D高斯点阵(GS)融合4D雷达与单目视觉信号的3D目标检测框架。3D GS将场景建模为连续高斯场,可动态分配资源至前景物体,兼具灵活性与效率。此外,4D雷达的速度维度提供运动线索,辅助锚定与优化高斯点的空间分布。RaGS采用级联流程构建并逐步精炼高斯场:首先通过基于视锥的定位初始化(FLI)将前景像素反投影,生成粗略高斯中心;随后迭代多模态聚合(IMA)显式利用图像语义、隐式融合4D雷达速度几何信息,精修感兴趣区域内的高斯点;最后多层级高斯融合(MGF)将高斯场渲染为分层BEV特征,用于3D目标检测。通过动态聚焦稀疏且信息丰富的区域,RaGS实现了以目标为中心的高精度与全面场景感知。在View-of-Delft、TJ4DRadSet和OmniHD-Scenes数据集上的大量实验验证了其鲁棒性与领先性能。代码将公开。
原文摘要 · Abstract (English)
4D millimeter-wave radar is a promising sensing modality for autonomous driving, yet effective 3D object detection from 4D radar and monocular images remains challenging. Existing fusion approaches either rely on instance proposals lacking global context or dense BEV grids constrained by rigid structures, lacking a flexible and adaptive representation for diverse scenes. To address this, we propose RaGS, the first framework that leverages 3D Gaussian Splatting (GS) to fuse 4D radar and monocular cues for 3D object detection. 3D GS models the scene as a continuous field of Gaussians, enabling dynamic resource allocation to foreground objects while maintaining flexibility and efficiency. Moreover, the velocity dimension of 4D radar provides motion cues that help anchor and refine the spatial distribution of Gaussians. Specifically, RaGS adopts a cascaded pipeline to construct and progressively refine the Gaussian field. It begins with Frustum-based Localization Initiation (FLI), which unprojects foreground pixels to initialize coarse Gaussian centers. Then, Iterative Multimodal Aggregation (IMA) explicitly exploits image semantics and implicitly integrates 4D radar velocity geometry to refine the Gaussians within regions of interest. Finally, Multi-level Gaussian Fusion (MGF) renders the Gaussian field into hierarchical BEV features for 3D object detection. By dynamically focusing on sparse and informative regions, RaGS achieves object-centric precision and comprehensive scene perception. Extensive experiments on View-of-Delft, TJ4DRadSet, and OmniHD-Scenes demonstrate its robustness and SOTA performance. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。