构建多模态数据集与框架，实现跨成像模态的高质量3D渲染。
MultimodalStudio: A Heterogeneous Sensor Dataset and Framework for Neural Rendering across Multiple Imaging Modalities
- 设计模块化框架，支持多种成像设备输入
- 在32个场景上验证跨模态信息迁移能力
- 适合从事多模态三维重建的研究者
神经辐射场（NeRF）在从任意视角渲染3D场景方面表现出色。尽管RGB图像常用于训练体渲染模型，但其他辐射模态的兴趣正日益增长。然而，隐式神经模型在异构成像模态间学习和迁移信息的能力尚未被充分探索，主要受限于训练数据稀缺。为此，我们提出MultimodalStudio（MMS）：包含MMS-DATA和MMS-FW。MMS-DATA是一个多模态多视角数据集，涵盖32个场景，使用5种成像模态（RGB、灰度、近红外、偏振、多光谱）采集。MMS-FW是一个新型模块化多模态NeRF框架，可处理多模态原始数据，并支持任意数量的多通道设备。通过大量实验，我们证明了在MMS-DATA上训练的MMS-FW能够实现不同成像模态间的知识迁移，生成的渲染质量优于单一模态。我们公开发布数据集与框架，以促进多模态体渲染及相关研究的发展。
原文摘要 · Abstract (English)
Neural Radiance Fields (NeRF) have shown impressive performances in the rendering of 3D scenes from arbitrary viewpoints. While RGB images are widely preferred for training volume rendering models, the interest in other radiance modalities is also growing. However, the capability of the underlying implicit neural models to learn and transfer information across heterogeneous imaging modalities has seldom been explored, mostly due to the limited training data availability. For this purpose, we present MultimodalStudio (MMS): it encompasses MMS-DATA and MMS-FW. MMS-DATA is a multimodal multi-view dataset containing 32 scenes acquired with 5 different imaging modalities: RGB, monochrome, near-infrared, polarization and multispectral. MMS-FW is a novel modular multimodal NeRF framework designed to handle multimodal raw data and able to support an arbitrary number of multi-channel devices. Through extensive experiments, we demonstrate that MMS-FW trained on MMS-DATA can transfer information between different imaging modalities and produce higher quality renderings than using single modalities alone. We publicly release the dataset and the framework, to promote the research on multimodal volume rendering and beyond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。