直接处理NeRF权重，让大模型理解3D几何与外观。
Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training
- 将NeRF的MLP权重直接输入大模型，跳过渲染过程。
- 在超30万条数据上训练，性能优于依赖2D/3D表示的方法。
- 适合做3D内容理解、跨模态生成的研究者使用。
多模态大语言模型（MLLM）在理解图像和3D数据方面取得显著进展，但这些模态在全面表征物体几何与外观方面存在固有局限。神经辐射场（NeRF）作为替代方案，通过简单多层感知机（MLP）的权重编码几何与逼真视觉属性。本文探索将NeRF纳入MLLM的可行性与有效性，提出首个可执行NeRF描述生成与问答任务的MLLM——LLaNA。该模型直接处理NeRF的MLP权重，无需渲染图像或构建3D结构即可提取对象信息。我们构建了首个大规模NeRF-语言数据集，包含超过30万条在ShapeNet与Objaverse上训练的NeRF及其配对文本注释，支持多种NeRF-语言任务。基于此数据集，我们建立评估基准。结果表明，直接处理NeRF权重在多项任务中表现优于依赖2D或3D表示的方法。
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities in understanding both images and 3D data, yet these modalities face inherent limitations in comprehensively representing object geometry and appearance. Neural Radiance Fields (NeRFs) have emerged as a promising alternative, encoding both geometric and photorealistic properties within the weights of a simple Multi-Layer Perceptron (MLP). This work investigates the feasibility and effectiveness of ingesting NeRFs into an MLLM. We introduce LLaNA, the first MLLM able to perform new tasks such as NeRF captioning and Q\&A, by directly processing the weights of a NeRF's MLP. Notably, LLaNA is able to extract information about the represented objects without the need to render images or materialize 3D data structures. In addition, we build the first large-scale NeRF-language dataset, composed by more than 300K NeRFs trained on ShapeNet and Objaverse, with paired textual annotations that enable various NeRF-language tasks. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that directly processing NeRF weights leads to better performance on NeRF-Language tasks compared to approaches that rely on either 2D or 3D representations derived from NeRFs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。