用近红外和元数据增强农业场景3D重建,提升光照不均下的精度。
Reconstruction Using the Invisible: Intuition from NIR and Metadata for Enhanced 3D Gaussian Splatting
- 融合近红外图像与文本元数据,通过交叉注意力建模多模态信息。
- 在复杂光照下实现优于3DGS等方法的重建效果,尤其在遮挡区域表现更优。
- 适合农业三维建模、植物健康监测等应用,开源数据集与代码可复现。
尽管3D高斯点阵(3DGS)发展迅速,其在农业领域的应用仍较有限。农业场景面临光照不均、遮挡严重和视场受限等挑战。为此,我们提出 extbf{NIRPlant}——一个包含近红外(NIR)图像、RGB图像、文本元数据、深度图及LiDAR数据的新型多模态数据集,覆盖多种室内外光照条件。通过引入NIR数据,方法增强了鲁棒性并提供可见光以外的植物生理信息。同时,利用基于植被指数(如NDVI、NDWI、叶绿素指数)的文本元数据,显著丰富了复杂农业环境的上下文理解。为充分融合多模态信息,我们提出 extbf{NIRSplat},一种基于交叉注意力机制与3D点位置编码的高斯点阵架构,提供强几何先验。大量实验表明, extbf{NIRSplat}在挑战性农业场景中优于3DGS、CoR-GS和InstantSplat等基准方法。代码与数据集已公开:https://github.com/StructuresComp/3D-Reconstruction-NIR
原文摘要 · Abstract (English)
While 3D Gaussian Splatting (3DGS) has rapidly advanced, its application in agriculture remains underexplored. Agricultural scenes present unique challenges for 3D reconstruction methods, particularly due to uneven illumination, occlusions, and a limited field of view. To address these limitations, we introduce \textbf{NIRPlant}, a novel multimodal dataset encompassing Near-Infrared (NIR) imagery, RGB imagery, textual metadata, Depth, and LiDAR data collected under varied indoor and outdoor lighting conditions. By integrating NIR data, our approach enhances robustness and provides crucial botanical insights that extend beyond the visible spectrum. Additionally, we leverage text-based metadata derived from vegetation indices, such as NDVI, NDWI, and the chlorophyll index, which significantly enriches the contextual understanding of complex agricultural environments. To fully exploit these modalities, we propose \textbf{NIRSplat}, an effective multimodal Gaussian splatting architecture employing a cross-attention mechanism combined with 3D point-based positional encoding, providing robust geometric priors. Comprehensive experiments demonstrate that \textbf{NIRSplat} outperforms existing landmark methods, including 3DGS, CoR-GS, and InstantSplat, highlighting its effectiveness in challenging agricultural scenarios. The code and dataset are publicly available at: https://github.com/StructuresComp/3D-Reconstruction-NIR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。